Publishing type：Research paper (scientific journal)   Publisher：Wiley  

Abstract

Background

While communication with dementia patients is challenging, known educational methods to improve communication skill for medical professionals are lacking. Our study aimed to assess the efficacy of simulated communication skills training for nursing students using augmented reality (AR) with real‐time feedback.

Methods

This is a randomized control study. Twenty‐five nursing students enrolled and learned standardized multimodal comprehensive care communication skills through a self‐learning material, which includes pathophysiology and clinical manifestations of dementia. Subsequently, participants were randomly assigned to one of the two learning systems—AR training or conventional nursing mannequin training. Each group had the training intervention to change the clothes of mannequins for one hour. The mannequin of the AR group was a superimposed computer graphic of an elderly woman’s face which reacted to participants' communication. Further, the communication skills of gaze and voice were evaluated by artificial intelligence (AI) and feedbacked to the participants’ head‐mounted display in real‐time. The conventional training group had the self‐training with nursing mannequins. All participants performed basic nursing care, including changing clothes and bed bath, to simulated patients before and after the training intervention, which were video‐recorded by an eye‐tracking camera and a fixed camera, then the videos were analyzed the communication skills by AI. Additionally, participants' empathy to patients was evaluated by the Jefferson Scale of Empathy‐Health Professions Students Version (JSPE‐HSP). The primary outcome was the proportion of time spent in eye contact during the care to simulated patients. The secondary outcome was the empathy score of JSPE‐HSP.

Results

After the training intervention, the proportion of time spent in eye contact in the AR training group significantly increased than the conventional training group (eye contact 13.6% versus 4.4%, P<0.05). Moreover, the JSPE‐HSP score increased from pre‐training to post‐training in the AR training group, whereas it decreased in the conventional training group [Mean (SD): 9.1 (6.6) versus ‐1.3 (3.8), P<0.01].

Conclusions

The simulated communication skills training for nursing students using augmented reality with real‐time feedback was associated with increased interactive communication skills to simulated patients and the empathy to patients.

DOI： 10.1002/alz.062055

researchmap

Publishing type：Research paper (international conference proceedings)   Publisher：ACM  

DOI： 10.1145/3503252.3533721

researchmap

Other Link： https://dblp.uni-trier.de/db/conf/um/umap2022.html#TorresenN22

Language：English   Publishing type：Research paper (scientific journal)  

DOI： 10.3389/frobt.2022.1020103

PubMed

researchmap

Publishing type：Research paper (scientific journal)  

We investigated the functions and mechanisms of non-goal-driven eye movements, which are defined as eye movements induced when looking at visual stimuli on a display without engaging in a specific task or looking at a display without any visual stimuli or tasks. In our experiment, participants were asked to perform a visual search task on a display, which was followed by a rest period in which stimuli remained on the display or all stimuli were erased. During the rest period, the participants were asked to only look at the displays without engaging in any visual or cognitive tasks. We mainly analyzed the gaze-shift patterns in both task and rest periods, in which eye movements were classified in accordance with the angles of saccade directions in two consecutive saccades. The results indicate a significant difference between goal-driven eye movements, which were observed in the task period, and nongoal-driven eye movements, which were observed in the rest period. Scanning gaze-shift patterns dominated the task period, and backward and corrective-saccade-like gaze-shift patterns dominated the rest period. The gaze-shift pattern was affected by the task-difficulty during the task period. From these findings, we propose a model describing the oculomotor system in terms of goal-driven and non-goal-driven eye movements. In this model, the engagement levels of top-down and bottom-up control change along with task difficulty and are affected by the gaze-shift patterns during a visual search task. Decoupling of top-down control from the oculomotor system during a rest period induces backward saccades, resulting in fixation around the central part of a display. Therefore, we suggest that non-goaldriven eye movements play a crucial role in maintaining the readiness of the oculomotor system for the next search task.

DOI： 10.16910/jemr.15.2.2

Scopus

researchmap

Publishing type：Research paper (international conference proceedings)   Publisher：IEEE  

DOI： 10.1109/RO-MAN53752.2022.9900706

researchmap

Other Link： https://dblp.uni-trier.de/db/conf/ro-man/ro-man2022.html#HaraTN22

Publishing type：Research paper (international conference proceedings)   Publisher：IEEE  

DOI： 10.1109/ICPR56361.2022.9956508

researchmap

Other Link： https://dblp.uni-trier.de/db/conf/icpr/icpr2022.html#TakeuchiN22

Publishing type：Research paper (scientific journal)   Publisher：Institute of Electronics, Information and Communications Engineers (IEICE)  

DOI： 10.1587/transinf.2020edp7218

researchmap

Language：English   Publishing type：Research paper (scientific journal)  

Due to cognitive and socio-emotional decline and mental diseases, senior citizens, especially people with dementia (PwD), struggle to interact smoothly with their caregivers. Therefore, various care techniques have been proposed to develop good relationships with seniors. Among them, Humanitude is one promising technique that provides caregivers with useful interaction skills to improve their relationships with PwD, from four perspectives: face-to-face interaction, verbal communication, touch interaction, and helping care receivers stand up (physical interaction). Regardless of advances in elderly care techniques, since current social robots interact with seniors in the same manner as they do with younger adults, they lack several important functions. For example, Humanitude emphasizes the importance of interaction at a relatively intimate distance to facilitate communication with seniors. Unfortunately, few studies have developed an interaction model for clinical care communication. In this paper, we discuss the current challenges to develop a social robot that can smoothly interact with PwDs and overview the interaction skills used in Humanitude as well as the existing technologies.

DOI： 10.3389/frobt.2021.650906

Scopus

PubMed

researchmap

Language：English   Publishing type：Research paper (scientific journal)   Publisher：TAYLOR & FRANCIS LTD  

In this paper, we investigate which approach to generate eye behaviors using an android robot makes what impressions on humans and clarify which are the important factors for attractive eye behaviors. Thus, we evaluate the human impression of eye behaviors displayed by an android robot while talking to a human by comparing the motions generated by the two approaches. In the first approach, we develop a method to imitate human eye behavior obtained from eye trackers. In the second approach, we control the eye direction, eye-contact duration, and eyeblinks based on the findings of human eye behavior in psychology and cognitive research. In the experiments, we asked male and female subjects to evaluate their impression by comparing the eye behaviors with an android that controls the eye-contact duration and eyeblinks by editing the imitation parameters or the rule-based behavior. In the experiments, we asked subjects to evaluate their impression of different eye behaviors displayed by an android. The eye behaviors were generated by modifying the imitation parameters or the rule-based behavior, which resulted in different eye-contact duration and eyeblink duration and timing. From the results, we found that (1) the imitation and rule-based behaviors showed no difference in terms of human-likeness, (2) the 3-second eye contact obtained better scores regardless of the imitation or rule-based eye behavior, (3) the subjects might regard the long eyeblinks as voluntary eyeblinks, with the intention to break eye contact, and (4) female subjects preferred short eyeblinks rather than long ones and considered that short eyeblinks might be one of the keys to making eye contact more suitable, in contrast to male subjects who preferred long eyeblinks.

DOI： 10.1080/01691864.2021.1928544

Web of Science

Scopus

researchmap

J-GLOBAL

researchmap

Publishing type：Research paper (scientific journal)  

The cornea in the human eye reflects incoming environmental light, which means we can obtain information about the surrounding environment from the corneal reflection in facial images. In recent years, as the quality of consumer cameras increases, this has caused privacy concerns in terms of identifying the people around the subject or where the photo is taken. This paper investigates the security risk of eye corneal reflection images: specifically, visual place recognition from eye reflection images. First, we constructed two datasets containing pairs of scene and corneal reflection images. The first dataset is taken in a virtual environment. We showed pre-captured scene images in a 180-degree surrounding display system and took corneal reflections from subjects. The second dataset is taken in an outdoor environment. We developed several visual place recognition algorithms, including CNN-based image descriptors featuring a naive Siamese network and AFD-Net combined with entire image feature representations including VLAD and NetVLAD, and compared the results. We found that AFD-Net+VLAD performed the best and was able to accurately determine the scene in 73.08% of the top-five candidate scenes. These results demonstrate the potential to estimate the location at which a facial picture was taken, which simultaneously leads to a) positive applications such as the localization of a robot while conversing with persons and b) negative scenarios including the security risk of uploading facial images to the public.

DOI： 10.1109/ACCESS.2021.3071406

Scopus

researchmap

Publishing type：Research paper (scientific journal)  

Gesture recognition and human-Activity recognition from multi-channel sensory data are important tasks in wearable and ubiquitous computing. In these tasks, increasing both the number of recognizable activity classes and the recognition accuracy is essential. However, this is usually an ill-posed problem because individual differences in the same gesture class may affect the discrimination of different gesture classes. One promising solution is to use personal classifiers, but this requires personal gesture samples for re-Training the classifiers. We propose a method of solving this issue that obtains personal gesture classifiers using few user gesture samples, thus, achieving accurate gesture recognition for an increased number of gesture classes without requiring extensive user calibration the novelty of our method is introducing a generative adversarial network (GAN)-based style transformer to 'generate' a user's gesture data the method synthesizes the gesture examples of the target class of a target user by transforming of a) gesture data into another class of the same user (intra-user transformation) or b) gesture data of the same class of another user (inter-user transformation) the synthesized data are then used to train the personal gesture classifier. We conducted comprehensive experiments using 1) different classifiers including SVM and CNN, 2) intra-and inter-user transformations, 3) various data-missing patterns, and 4) two different types of sensory data. Results showed that the proposed method had an increased performance. Specially, the CNN-based classifiers increased in average accuracy from 0.747 to 0.822 in the CheekInput dataset and from 0.856 to 0.899 in the USC-HAD dataset. Moreover, the experimental results with various data-missing conditions revealed a relation between the number of missing gesture classes and the accuracy of the existing and proposed methods, and we were able to clarify several advantages of the proposed method these results indicate the potential of considerably reducing the number of required training samples of target users.

DOI： 10.1145/3432199

Scopus

researchmap

Publishing type：Research paper (international conference proceedings)   Publisher：IEEE  

In this paper, we consider Simulated and Unsupervised (S+U) learning which is a problem of learning from labeled synthetic and unlabeled real images. After translating the synthetic images to real ones, existing S+U learning methods use only the labeled synthetic images for training a predictor (e.g., a regression function) and ignore the target real images, which may result in unsatisfactory prediction performance. Our approach utilizes both synthetic and real images to train the predictor. The main idea of ours is to involve a self-ensemble learning framework into S+U learning. More specifically, we require the prediction results for an unlabeled real image to be consistent between 'teacher' and 'student' predictors, even after some perturbations are added to the image. Furthermore, aiming at generating diverse perturbations along the underlying data manifold, we introduce one-to-many image translation between synthetic and real images. Evaluation experiments on an appearance-based gaze estimation task demonstrate that the proposed ideas can improve the prediction accuracy and our full method can outperform existing S+U learning methods.

DOI： 10.1109/ICIP40778.2020.9191100

Scopus

researchmap

Other Link： https://dblp.uni-trier.de/db/conf/icip/icip2020.html#MitsuzumiIKN20

Publishing type：Research paper (scientific journal)  

In this paper, we describe a wearable first-person video (FPV) analysis system for evaluating the skill levels of caregivers. This is a part of our project that aims to quantize and analyze the tender-care technique known as Humanitude by using wearable sensing and AI technology devices. Using our system, caregivers can evaluate and elevate their care levels by themselves. From the FPVs of care sessions taken by wearable cameras worn by caregivers, we obtained the 3D facial distance, pose and eye-contact states between caregivers and receivers by using facial landmark detection and deep neural network (DNN)-based eye contact detection. We applied statistical analysis to these features and developed algorithms that provide scores for tender-care skill. In experiments, we first evaluated the performance of our DNN-based eye contact detection by using eye contact datasets prepared from YouTube videos and FPVs that assume conversational scenes. We then performed skill evaluations by using Humanitude training scenes involving three novice caregivers, two Humanitude experts and seven middle-level students. The results showed that our eye contact detection outperformed existing methods and that our skill evaluations can estimate the care skill levels.

DOI： 10.1007/s10846-019-01052-8

Scopus

researchmap

Publishing type：Research paper (scientific journal)  

Objective: Conventional surgical assistance and skill analysis for suturing mostly focus on the motions of the tools. As the quality of the suturing is determined by needle motions relative to the tissues, having knowledge of the needle motion would be useful for surgical assistance and skill analysis. As the first step toward demonstrating the usefulness of the knowledge of the needle motion, we developed a needle detection algorithm. Methods: Owing to the small needle size, attaching sensors to it is difficult. Therefore, we developed a real-time video-based needle detection algorithm using a region-based convolutional neural network. Results: Our method successfully detected the needle with an average precision of 89.2%. The needle was robustly detected even when the needle was heavily occluded by the tools and/or the blood vessels during microvascular anastomosis. However, there were some incorrect detections, including partial detection. Conclusion: To the best of our knowledge, this is the first time deep neural networks have been applied to real-time needle detection. In the future, we will develop a needle pose estimation algorithm using the predicted needle location toward computer-aided surgical assistance and surgical skill analysis.

DOI： 10.1007/s11548-019-02050-9

Scopus

PubMed

researchmap

Other Link： https://dblp.uni-trier.de/db/journals/cars/cars15.html#NakazawaHMJ20

Publishing type：Research paper (international conference proceedings)   Publisher：Springer  

Eye contact is fundamental for human communication and social interactions; therefore much effort has been made to develop automated eye-contact detection using image recognition techniques. However, existing methods use first-person-videos (FPV) that need participants to equip wearable cameras. In this work, we develop an novel eye contact detection algorithm taken from normal viewpoint (third person video) assuming the scenes of conversations or social interactions. Our system have high affordability since it does not require special hardware or recording setups, moreover, can use pre-recorded videos such as Youtube and home videos. In designing algorithm, we first develop DNN-based one-sided gaze estimation algorithms which output the states whether the one subject looks at another. Afterwards, eye contact is found at the frame when the pair of one-sided gaze happens. To verify the proposed algorithm, we generate third-person eye contact video dataset using publicly available videos from Youtube. As the result, proposed algorithms performed 0.775 in precision and 0.671 in recall, while the existing method performed 0.484 in precision and 0.061 in recall, respectively.

DOI： 10.1007/978-3-030-41299-9_52

Scopus

researchmap

Other Link： https://dblp.uni-trier.de/db/conf/acpr/acpr2019-2.html#OhshimaN19

Language：Japanese   Publisher：The Japan Society of Mechanical Engineers  

Toward the achievement of natural attentive listening using an android, a nodding synchrony behavior with a human talker is an effective nonverbal behavior from the finding of several works. In addition, according to the related work, a human listener’s eyeblinks synchronizes with an android talker’s one in face-to-face communication. Here, we hypothesize if an android which is in the role of a listener imitates eyeblinks and nodding simultaneously a human’s one in face-to-face communication, the android can act more attentive listening behavior. In this paper, we propose a method to imitate human eyeblinks and nodding using an android to synchronize the human’s one. Through the experiment, we confirmed that the android can generate the imitation behaviors of human eyeblinks and nodding.

DOI： 10.1299/jsmermd.2020.1p2-e16

CiNii Article

researchmap

Publishing type：Research paper (international conference proceedings)   Publisher：Springer  

Gaze behavior, including eye contact and gaze direction, is an essential component of non-verbal communication, helping to facilitate human-to-human conversation in ways that have often been thought of as universal and innate. However, these have been shown to be influenced partially by cultural norms and background, and despite this, the majority of social robots do not have any cultural-based non-verbal behaviors and several lack any directional gaze capabilities at all. This study aims to observe how different gaze behaviors manifest during conversation as a function of culture as well as exposure to other cultures, by examining differences in behaviors such as duration of direct gaze, duration and direction of averted gaze, and average number of shifts in gaze, with the objective of establishing a baseline of Japanese gaze behavior to be implemented into a social robot. Japanese subjects were found to have much more averted gaze during a task that involves thinking as opposed to a task focused on communication. Subjects with significant experience living overseas were found to have different directional gaze patterns from subjects with little to no overseas experience, implying that non-verbal behavior patterns can change with exposure to other cultures.

DOI： 10.1007/978-3-030-49570-1_33

Scopus

researchmap

Other Link： https://dblp.uni-trier.de/db/conf/hci/hci2020-14.html#HardjasaN20

Publishing type：Research paper (international conference proceedings)   Publisher：Springer  

It is known that a person, when ignored by others, causes various negative reactions and enhances aggressive behavior and Self-destructive behavior [1–3]. The same goes for elderly people with dementia, therefore, depending on the care may cause fear and confusion for elderly people with dementia [4]. At the same time, care for the elderly with dementia has a great mental and physical burden on caregivers. For this reason, the turnover rate has increased and it has become difficult to provide adequate care [5–8]. For these problems, as one of the dementia care is gaining attention Humanitude [4, 10]. Humanitude consists of four skills: “see”, “touch”, “speak”, and “stand” [9–11]. So, our research focus on “see”, one of the basic skills. A person’s gaze is generally directed to an object of interest or attention. The gaze is extremely useful information for estimating the mind of another person [3, 12, 13]. Therefore, we let a caregiver wear the first person camera, and four types of “seeing” behavior patterns during the oral care (care receiver → caregiver, care receiver ← caregiver, mutual gaze, none) measured. We compared the differences between Humanitude experts and novice. There was a large difference in the frequency and time of mutual gaze between expert and novice caregiver for care receiver. The act of matching the sight of eyes is an act of not ignoring the other person, indicating an interest in the care receiver, and it is considered that for the care receiver, the anxiety and fear during care are reduced.

DOI： 10.1007/978-3-030-50249-2_7

Scopus

researchmap

Other Link： https://dblp.uni-trier.de/db/conf/hci/hci2020-28.html#IwamotoN20

Publishing type：Research paper (international conference proceedings)   Publisher：IEEE  

The appearance of android robots is very similar to that of human beings. From their appearance, we expect that androids might provide us with high-level communication. The imitation of human behavior gives us the feeling of natural behavior even if we do not know what drives high-level communication. In this paper, we evaluate the imitation of human eye behavior by an android. We consider that the android imitates human eye behavior while explaining some research topic and a person acts as a listener. Then, we construct a method to imitate the eye behavior obtained from eye trackers. For the evaluation, we asked seventeen male subjects for their subjective evaluation and compared the imitation with an android that controlled eye-contact duration and eyeblinks by editing the imitation or programming rule-based behavior. From the results, we found out that 1) the rule-based behaviors kept human-likeness, 2) 3-second eye contact obtained better scores regardless of the imitation-based or rule-based eye behavior, and 3) the subjects might regard the longer eyeblinks as voluntary eyeblinks, with the intention to break eye contacts.

DOI： 10.1109/RO-MAN46459.2019.8956387

Scopus

researchmap

Other Link： https://dblp.uni-trier.de/db/conf/ro-man/ro-man2019.html#SanoYRTNO19

Publishing type：Research paper (international conference proceedings)   Publisher：IEEE  

In this paper, we describe a wearable first-person video (FPV) analysis system for evaluating the skill levels of tender dementia-care technique. Using this system, caregivers can evaluate and elevate their care levels by themselves using the systems' feedbacks. From the FPVs of care sessions taken by wearable cameras worn by caregivers, we obtained the 3D facial distance, pose and eye-contact states between caregivers and receivers by using facial landmark detection and deep neural network (DNN)-based eye contact detection. We applied statistical analysis to these features and developed algorithms that provide scores for tender-care skill. To find and confirm our idea, we conducted chronological study to observe the progression of tender care-skill learning using care learners. First, we took FPVs while care training scenes involving novice caregivers, tender-care experts and middle-level students, and found major behavioural differences among them. Second, we performed the same experiments for the participants before and after training sessions of the care. As the result, we found the same behavioural difference between 1) novices and experts and 2) novices before and after taking training sessions. These results indicate that our FPV-based behavior analysis can evaluate the skill progression of the tender dementia-care.

DOI： 10.1109/ICCVW.2019.00544

Scopus

researchmap

Other Link： https://dblp.uni-trier.de/db/conf/iccvw/iccvw2019.html#NakazawaH19

Publishing type：Research paper (scientific journal)  

The diagnosis of ocular misalignment is difficult and needs examination by ophthalmologists and orthoptists. However, there are not enough qualified personnel to perform such diagnoses. The eye position check is in part systematized. With this check system, we can detect not only the symptoms but also the angle and the extent of strabismus. However, the types of strabismus that can be detected with this technique are limited to exotropia. The purpose of this study is to develop a simplified check system to screen at least the presence of strabismus apart from the type of strabismus or amount of ocular deviation. First, we digitalized the check process. Specifically, we digitized the elemental technology, i.e., the cover–uncover function, required for automation of the typical cover test for eye position check. Furthermore, we developed and implemented an abnormality determination process and evaluated the performance of the system through this experiment, the results of which indicated a higher detection capability than the conventional cover test performed by ophthalmologists and orthoptists.

DOI： 10.1007/s10015-018-0520-4

Scopus

researchmap

Publishing type：Research paper (international conference proceedings)   Publisher：IEEE  

In production lines, human workers assemble and/or inspect products in a predetermined process flow. Training new workers is a complex process with varying results. To track the workers motion in a way to better understand human skill, an overhead semantic part segmentation of workers is desired. For this purpose, in this work, we propose a fully-convolutional neural-network model paired with four proposed augmentation strategies. Artificial depth images were used as training data and the augmentation strategies were essential in aiding the network to generalize to the real images. The proposed method was evaluated in two tasks with different backgrounds: a part assembly task and a quality check task. We improved the F-measure by 12% in the part assembly task and 4% in the quality check task when compared to our previous work.

DOI： 10.1109/CBS46900.2019.9114417

Scopus

researchmap

Other Link： https://dblp.uni-trier.de/db/conf/cbs/cbs2019.html#YoshimuraMNHMMN19

Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：IEEE  

In this paper, we present a robust pupil detection method from visible light (VL) images using a convolutional neural network (CNN). In contrast to existing pupil detection algorithms, our method does not require infrared (IR) illuminations and cameras, and robustly works even for the images of dark-brown irises (black eyes) and/or images including strong corneal reflections such as outdoor scenes. Thus it has potential application scenarios. Our method first detects an eye region from an input image and applies CNN-based pupil segmentation which has a composition and decomposition structure. To learn the relationship between visible eye images and pupil segments, we construct two datasets and augmentation algorithms. One dataset is based on an existing eye image dataset (UBIRIS.v2), and the other consists of images taken by ourselves by using a corneal imaging camera that can take eye images in mobile environments. Applying color and corneal reflection augmentations to these image datasets and using them for learning the CNN, we built a robust pupil segmentation neural network. The performance is evaluated in three ways. First, we evaluate the segmentation accuracy. Second, we evaluate the pupil center detection accuracy using GI4E facial image sets. Third, we developed an eye gaze tracking (EGT) algorithm that uses the pupil detection and evaluated its accuracy. From the result, the proposed method detects pupil centers more accuracy than the state-of-the arts, and shows similar EGT accuracy to the commercial systems that use IR-active lighting setups.

DOI： 10.1109/SMC.2018.00154

Scopus

researchmap

Other Link： https://dblp.uni-trier.de/db/conf/smc/smc2018.html#KitazumiN18

Publishing type：Research paper (international conference proceedings)   Publisher：IEEE  

Eye contact (mutual gaze) is a foundation of human communication and social interactions; therefore, it is studied in many fields such as psychology, social science, and medicine. Our group have been studied wearable vision-based eye contact detection techniques using a first person camera for the purpose of evaluating the gaze skills in the tender dementia care. In this work, we search for deep learning-based eye contact detection techniques from small number of labeled images. We implemented and tested two eye contact detection algorithms: naïve deep-learning-based algorithm and generative adversarial networks (GAN)-based semi supervised learning (SSL) algorithm. These methods are learned and verified by using Columbia Gaze Dataset, Facescrub and our original datasets. The results show the effectiveness and limitations of the deep-learning-based and GAN-based approaches. Interestingly, we found the bilateral difference of the accuracy of eye contact detection with respect to the facial pose with respect to the camera, which is expected to be caused by the learning datasets.

DOI： 10.1109/SMC.2018.00666

Scopus

researchmap

Other Link： https://dblp.uni-trier.de/db/conf/smc/smc2018.html#MitsuzumiN18

Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：IEEE  

Eye contact (mutual gaze) is fundamental for human communication and social interactions; therefore, it is studied in many fields. To support the study of eye contact, much effort has been made to develop automated eye-contact detection using image recognition techniques. In recent years, convolutional neural network (CNN) based eye-contact detection techniques are becoming popular due to their performance; however, they mainly use single frame for recognition. Eye contact is a human communication behavior, so temporal information, such as temporal eye images and facial poses, is important to increase the accuracy of eye-contact detection. We incorporate temporal information into eye-contact detection by using temporal neural network structures that combine CNNs and long short-term memory (LSTM). We tested several network combinations of CNNs and LSTM and found the best solution that uses the outputs of CNNs as well as the cell state vectors of LSTM in the fully connected layers. We prepared two types of eye contact video datasets. One dataset is based on online videos, and the other was taken by a first-person camera in assumed conversational scenarios. The results show that our method is better than the approaches that use single frames. Namely, our method performs 0.8781, while the existing method (DeepEC) performed 0.8319, in F-1-score.

DOI： 10.23919/MVA.2019.8757989

Web of Science

Scopus

researchmap

Other Link： https://dblp.uni-trier.de/db/conf/mva/mva2019.html#WatanabeNMN19

Language：English   Publishing type：Research paper (scientific journal)   Publisher：INST IMAGE INFORMATION & TELEVISION ENGINEERS  

Separating reflection components is a fundamental problem in computer vision and useful for many applications such as image quality. We propose a novel method that improves the accuracy of separating reflection components from a single image. Although several algorithms for separating reflection components have been proposed, our method can additionally improve the accuracy based on their results. First, we obtain diffuse and specular components by using an existing algorithm. Then, we apply a high-emphasis filter for each component. Since the responses of the high-emphasis filter where the separation fails become larger than the original values, we can detect erroneous pixels. Thus, we replace separation results of these erroneous pixels with results of other reference pixels from the image considering the similarity between the target and reference pixels. Experimental results show that our method can improve at most 13.61 dB in terms of the Peak Signal-to-Noise Ratio (PSNR).

DOI： 10.3169/mta.7.92

Web of Science

Scopus

researchmap

Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：IEEE  

The fashion style recognition is important in online marketing applications. Several algorithms have been proposed, but their accuracy is still unsatisfactory. In this paper, we share our proposed method for creating an improved fashion style recognition algorithm, component-dependent convolutional neural networks (CD-CNNs). Given that a lot of fashion styles largely depend on the features of specific body parts or human body postures, first, we obtain images of the body parts and postures by using semantic segmentation and pose estimation algorithms; then, we pre-train CD-CNNs. We perform the classification by the concatenated outputs of CD-CNNs and a support vector machine (SVM). Experimental results using the HipsterWars and FashionStyle14 datasets prove that our method is effective and can improve classification accuracy, namely 85.3% for HipsterWars and 77.7% for FashionStyle14, while those of existing methods were 80.9% for HipsterWars and 72.0% for FashionStyle14.

DOI： 10.1109/ICIP.2019.8803622

Web of Science

Scopus

researchmap

Other Link： https://dblp.uni-trier.de/db/conf/icip/icip2019.html#YamamotoN19

Language：Japanese  

researchmap

Publishing type：Research paper (scientific journal)  

Aims: Our aims were to develop a training system for camera assistants (CA), and evaluate participants' performance as CA. Methods: A questionnaire on essential requirements to be a good CA was administered to experts in pediatric endoscopic surgery. An infant-sized box trainer with several markers and lines inside was developed. Participants performed marker capturing and line-tracing tasks using a 5-mm 30° scope. A postexperimental questionnaire on the developed system was administered. The task completion time was measured. Results: The 5-point evaluation scale was used for each item in the questionnaire survey of experts. The abilities to maintain a horizontal line (mean score: 4.5) and to center the target in a specified rectangle on the monitor (4.5) as well as having a full understanding of the operative procedure (4.3) were ranked as highly important. Fifty-two participants, including 5 surgical residents, were enrolled in the evaluation experiment. The completion time of capturing the markers was significantly longer in the resident group than in the nonresident group (244 versus 124 seconds, P = .04), but that of tracing the lines was not significantly different between the groups. The postexperimental questionnaire showed that the participants felt that the line-tracing tasks (3.7) were more difficult than marker-capturing tasks (2.9). Conclusions: Being proficient in manipulating a camera and having adequate knowledge of operative procedures are essential requirements to be a good CA. The ability was different between the resident and nonresident groups even in a simple task such as marker capturing.

DOI： 10.1089/lap.2017.0406

Scopus

PubMed

researchmap

Language：English   Publishing type：Research paper (international conference proceedings)  

researchmap

Language：English   Publishing type：Research paper (international conference proceedings)  

researchmap

Language：Japanese   Publishing type：Research paper (conference, symposium, etc.)  

researchmap

Publishing type：Research paper (international conference proceedings)   Publisher：SPIE  

DOI： 10.1117/12.2292418

researchmap

Other Link： https://dblp.uni-trier.de/db/conf/miigp/miigp2018.html#ChenMKNDHM18

Language：English   Publishing type：Research paper (scientific journal)   Publisher：Institute of Electronics, Information and Communication, Engineers, IEICE  

We present a human point of gaze estimation system using corneal surface reflection and omnidirectional image taken by spherical panorama cameras, which becomes popular recent years. Our system enables to find where a user is looking at only from an eye image in a 360° surrounding scene image, thus, does not need gaze mapping from partial scene images to a whole scene image that are necessary in conventional eye gaze tracking system. We first generate multiple perspective scene images from an omnidirectional (equirectangular) image and perform registration between the corneal reflection and perspective images using a corneal reflection-scene image registration technique. We then compute the point of gaze using a corneal imaging technique leveraged by a 3D eye model, and project the point to an omnidirectional image. The 3D eye pose is estimate by using the particle-filter-based tracking algorithm. In experiments, we evaluated the accuracy of the 3D eye pose estimation, robustness of registration and accuracy of PoG estimations using two indoor and five outdoor scenes, and found that gaze mapping error was 5.546 [deg] on average.

DOI： 10.1587/transinf.2017MVP0020

Scopus

researchmap

Other Link： https://dblp.uni-trier.de/db/journals/ieicet/ieicet101d.html#OgawaNN18

Language：Japanese   Publishing type：Research paper (conference, symposium, etc.)  

researchmap

Language：Japanese   Publishing type：Research paper (conference, symposium, etc.)  

researchmap

Language：English   Publishing type：Research paper (international conference proceedings)  

researchmap

Language：English   Publishing type：Research paper (scientific journal)  

researchmap

Other Link： https://kaken.nii.ac.jp/grant/KAKENHI-PROJECT-17H01779/

Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：Institute of Electrical and Electronics Engineers Inc.  

We present a human point of gaze estimation system using corneal surface reflection and omni-directional image taken by a fish eye. Only capturing an eye image, our system entables to find where a user is looking in 360P surrounding scene image. We first generates multiple perspective scene images from an equi-rectangular image and perform registration between corneal reflection and perspective images. We then compute the point of gaze using a 3D eye model and project the point to an omni-directional image. We evaluated the robustness of registration and accuracy of PoG estimations using two indoor and five outdoor scenes, and found that gaze mapping error was 5.526[deg] on average. This result shows the potential to the marketing and outdoor training system.

DOI： 10.23919/MVA.2017.7986842

Scopus

researchmap

Language：Japanese   Publisher：株式会社医学書院  

DOI： 10.11477/mf.1429200917

researchmap

Other Link： https://kaken.nii.ac.jp/grant/KAKENHI-PROJECT-17H01779/

Language：English   Publishing type：Research paper (scientific journal)   Publisher：TAYLOR & FRANCIS LTD  

This paper shows a novel eye gaze tracking (EGT) technique using the corneal imaging technique. Compared to the existing pupil center and pupil reflection techniques, our approach directly finds PoG in the reflected scene image at the human corneal surface. As a result, it does not suffer from the parallax issue and does not require per-setup system calibrations. To achieve this system, we develop following techniques: First, we use the idea of the gaze-reflection point (GRP), where light from the PoG in the scene reflects at the corneal surface into an eye image. Second, illuminating the whole scene or particular objects using coded structured light enables robust and accurate matching at the GRP to obtain the PoG in a scene image. For this purpose, we show two implementations: a special high-power IR LED-array projector and active LED markers. Experimental evaluation shows that the proposed scheme achieves considerable accuracy and successfully supports depth-varying environments as well as practical applications including observation in a conversation scene. We believe the proposed EGT technique has considerably large potential to solve major issues of current EGT systems, expands the application fields of the EGT and increases the usability of interactive systems.

DOI： 10.1080/01691864.2016.1277552

Web of Science

researchmap

Other Link： https://dblp.uni-trier.de/db/journals/ar/ar31.html#NakazawaKNN17

Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：ELSEVIER SCIENCE BV  

The present authors have been developing a master-slave neurosurgical robot and its intelligent control for tasks in the deep and narrow spaces of the brain. This paper proposes a robotic autonomous control method for avoiding possible collisions between the shaft of a surgical robotic instrument and the surrounding tissues. To this end, a new robotic simulator was developed and used to evaluate the proposed method. The results showed the proof of concept of the proposed autonomous collision avoidance, which has the potential to enhance the safety of robotic neurosurgery in deep and narrow spaces. (C) 2016 The Authors. Published by Elsevier B.V.

DOI： 10.1016/j.procir.2017.04.027

Web of Science

researchmap

Language：English   Publishing type：Research paper (scientific journal)   Publisher：OPTICAL SOC AMER  

This paper introduces an image registration algorithm between an eye reflection and a scene image. Although there are currently a large number of image registration algorithms, this task remains difficult due to nonlinear distortions at the eye surface and large amounts of noise, such as iris texture, eyelids, eyelashes, and their shadows. To overcome this issue, we developed an image registration method combining an aspherical eye model that simulates nonlinear distortions considering eye geometry and a two-step iterative registration strategy that obtains dense correspondence of the feature points to achieve accurate image registrations for the entire image region. We obtained a database of eye reflection and scene images featuring four subjects in indoor and outdoor scenes and compared the registration performance with different asphericity conditions. Results showed that the proposed approach can perform accurate registration with an average accuracy of 1.05 deg by using the aspherical cornea model. This work is relevant for eye image analysis in general, enabling novel applications and scenarios. (C) 2016 Optical Society of America

DOI： 10.1364/JOSAA.33.002264

Web of Science

researchmap

Publishing type：Research paper (international conference proceedings)  

Robotic assistance enables a surgeon to perform dexterous and precise manipulations. However, conducting robot assisted neurosurgery within the deep and narrow spaces of the brain presents the risk of unexpected collisions between the shafts of robotic instruments and their surroundings out of the microscopic view. Thus, we propose the provision of feedback using a truncated cone shaped virtual fixture generated by marking the edges of the top and bottom plane of a workspace in the deep and narrow spaces within the brain with the slave manipulator. The experimental results show that the virtual fixture generation method could precisely model the workspace. We also implemented force feedback, visual feedback, and motion scaling feedback in the microsurgical robotic system in order to inform the surgeon of the risk of collision. Performance of each feedback method and their combinations was evaluated in two experiments. The experimental results showed that the combination of the force and the visual feedback methods were the most beneficial for avoiding collisions.

DOI： 10.1109/BIOROB.2016.7523632

Scopus

researchmap

Language：English  

DOI： 10.1007/978-3-319-15702-3_2

researchmap

Language：English  

The cornea of the human eye acts as a mirror that reflects light from a person's environment. These corneal reflections can be extracted from an image of the eye by modeling the eye-camera geometry as a catadioptric imaging system. As a result, one obtains the visual information of the environment and the relation to the observer (view, gaze), which allows for application in a number of fields. The recovered illumination map can be further applied to various computational tasks. This paper provides a comprehensive introduction on corneal imaging, and aims to show the potential of the topic and encourage advancement. It makes a number of contributions, including (1) a combined view on previously unrelated fields, (2) an overview of recent developments, (3) a detailed explanation on anatomic structures, geometric eye and corneal reflection modeling including multiple eye images, (4) a summary of our work and contributions to the field, and (5) a discussion of implications and promising future directions. The idea behind this paper is a geometric framework to solve persisting technical problems and enable non-intrusive interfaces and smart sensors for traditional, ubiquitous and ambient environments. © 2013 Information Processing Society of Japan.

DOI： 10.2197/ipsjtcva.5.1

Scopus

CiNii Article

researchmap

Language：English   Publisher：Information and Media Technologies Editorial Board  

The cornea of the human eye acts as a mirror that reflects light from a person's environment. These corneal reflections can be extracted from an image of the eye by modeling the eye-camera geometry as a catadioptric imaging system. As a result, one obtains the visual information of the environment and the relation to the observer (view, gaze), which allows for application in a number of fields. The recovered illumination map can be further applied to various computational tasks. This paper provides a comprehensive introduction on corneal imaging, and aims to show the potential of the topic and encourage advancement. It makes a number of contributions, including (1) a combined view on previously unrelated fields, (2) an overview of recent developments, (3) a detailed explanation on anatomic structures, geometric eye and corneal reflection modeling including multiple eye images, (4) a summary of our work and contributions to the field, and (5) a discussion of implications and promising future directions. The idea behind this paper is a geometric framework to solve persisting technical problems and enable non-intrusive interfaces and smart sensors for traditional, ubiquitous and ambient environments.

DOI： 10.11185/imt.8.389

CiNii Article

researchmap

Language：English   Publishing type：Research paper (scientific journal)   Publisher：SPRINGER  

We present a novel off-line algorithm for target segmentation and tracking in video. In our approach, video data is represented by a multi-label Markov Random Field model, and segmentation is accomplished by finding the minimum energy label assignment. We propose a novel energy formulation which incorporates both segmentation and motion estimation in a single framework. Our energy functions enforce motion coherence both within and across frames. We utilize state-of-the-art methods to efficiently optimize over a large number of discrete labels. In addition, we introduce a new ground-truth dataset, called Georgia Tech Segmentation and Tracking Dataset (GT-SegTrack), for the evaluation of segmentation accuracy in video tracking. We compare our method with several recent on-line tracking algorithms and provide quantitative and qualitative performance comparisons.

DOI： 10.1007/s11263-011-0512-5

Web of Science

researchmap

Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：SPRINGER-VERLAG BERLIN  

Eye gaze tracking (EGT) is a common problem with many applications in various fields. While recent methods have achieved improvements in accuracy and usability, current techniques still share several limitations. A major issue is the need for external calibration between the gaze camera system and the scene, which commonly restricts to static planar surfaces and leads to parallax errors. To overcome these issues, the paper proposes a novel scheme that uses the corneal imaging technique to directly analyze reflections from a scene illuminated with structured light. This comprises two major contributions: First, an analytic solution is developed for the forward projection problem to obtain the gaze reflection point (GRP), where light from the point of gaze (PoG) in the scene reflects at the corneal surface into an eye image. We also develop a method to compensate for the individual offset between the optical axis and true visual axis. Second, introducing active coded illumination enables robust and accurate matching at the GRP to obtain the PoG in a scene image, which is the first use of this technique in EGT and corneal reflection analysis. For this purpose, we designed a special high-power IR LED-array projector. Experimental evaluation with a prototype system shows that the proposed scheme achieves considerable accuracy and successfully supports depth-varying environments.

DOI： 10.1007/978-3-642-33709-3_12

Web of Science

researchmap

Language：English   Publishing type：Research paper (scientific journal)   Publisher：ACADEMIC PRESS INC ELSEVIER SCIENCE  

In this paper, we describe a novel method for calibrating display-camera setups from reflections in a user's eyes. Combining both devices creates a capable controlled illumination system that enables a range of interesting vision applications in non-professional environments, including object/face reconstruction and human computer interaction. One major issue barring such systems from average homes is the geometric calibration to obtain the pose of the display which requires special hardware and tedious user interaction. Our proposed approach eliminates this requirement by introducing the novel idea of analyzing screen reflections in the cornea of the human eye, a mirroring device that is always available. We employ a simple shape model to recover pose and reflection characteristics of the eye. Thorough experimental evaluation shows that the basic strategy results in a large error and discusses possible reasons. Based on the findings, a non-linear optimization strategy is developed that exploits geometry constraints within the system to considerably improve the initial estimate. It further allows to automatically resolve an inherent ambiguity that arises in image-based eye pose estimation. The strategy may also be integrated to improve spherical mirror calibration. We describe several comprehensive experimental studies which show that the proposed method performs stably with respect to varying subjects, display poses, eye positions, and gaze directions. The results are feasible and should be sufficient for many applications. In addition, the findings provide general insight on the application of eye reflections for geometric reconstruction. (C) 2011 Elsevier Inc. All rights reserved.

DOI： 10.1016/j.cviu.2011.02.008

Web of Science

researchmap

Language：Japanese   Publisher：The Institute of Image Information and Television Engineers  

DOI： 10.3169/itej.64.1830

CiNii Article

CiNii Books

researchmap

Other Link： https://jlc.jst.go.jp/DN/JALC/00360473643?from=CiNii

Language：Japanese   Publishing type：Research paper (scientific journal)   Publisher：Inst. of Image Information and Television Engineers  

DOI： 10.3169/itej.63.1224

Scopus

researchmap

Language：English   Publishing type：Research paper (scientific journal)   Publisher：IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG  

In this paper, we present a novel approach to recognize motion styles and identify people using the Multi Factor Tensor (MFT) model. We apply a musical information analysis method in segmenting the motion sequence relevant to the keyposes and the musical rhythm. We define a task model by considering the repeated motion segments, where the motion is decomposed into a person-invariant factor task and a person-dependent factor style. Given the motion data set, we formulate the MFT model, factorize it efficiently in different modes, and use it in recognizing the tasks and the identities of the persons performing the tasks. We capture the motion data of different people for a few cycles, segment it using the musical analysis approach, normalize the segments using a vectorization method, and realize our MFT model. In our experiments, Japanese traditional dance sequences performed by several people are used. Provided with an unknown motion segment which is to be probed and which was performed at a different time in the time space, we first normalize the motion segment and flatten our MFT model appropriately, then recognize the task and the identity of the person. We follow two approaches in conducting our experiments. In one experiment, we recognize the tasks and the styles by maximizing a function in the tensor subdomain, and in the next experiment, we use a function value in the tensorial subdomain with a threshold for recognition. Interestingly, unlike the first experiment, we are capable of recognizing tasks and human identities that were not known beforehand. We conducted various experiments to evaluate the potential of the recognition ability of our proposed approaches, and the results demonstrate the high accuracy of our model.

DOI： 10.1093/ietisy/e91-d.5.1531

Web of Science

CiNii Article

researchmap

Language：Japanese   Publisher：一般社団法人情報処理学会  

This paper shows the configurations and evaluations of the educational computer system in our university that serves variety types of applications for many numbers of users, reducing Total Cost of Ownership (TCO). As for client operating systems, we developed newly designed diskless Linux. The OS and other applications are loaded from servers through networks; therefore the clients don't have hard drives. This scheme drastically reduces TCO because it is not necessary to replace hard drives which fail most frequently in these types of systems. Also, the software updates of the clients are achieved just by uploading the files in few servers. This also reduces the maintenance cost rather than updating the software stored in local hard drives of many clients. As for the client application software, we installed not only OpenOffice.org or StarSuite, but also Microsoft Office combined with Crossover Office emulator that is highly demanded by end users.

CiNii Article

CiNii Books

J-GLOBAL

researchmap

Language：English   Publishing type：Part of collection (book)   Publisher：Springer US  

We propose a method which refines the range measurement of range finders by computing correspondences of vertices of multiple range images acquired from various viewpoints. Our method assumes that a range image acquired by a laser range finder has anisotropic error distribution which is parallel to the ray direction. Thus, we find corresponding points of range images along with the ray direction. We iteratively converge range images to minimize the distance of corresponding points. We describe the effectiveness of our method by the presenting the experimental results of artificial and real range data. Also we show that our method refines a 3D shape more accurately as opposed to that achieved by using the Gaussian filter. © 2008 Springer-Verlag US.

DOI： 10.1007/978-0-387-75807_11

Scopus

researchmap

Language：English   Publishing type：Part of collection (book)   Publisher：Springer US  

This chapter describes a fast, simultaneous alignment method for a large number of range images. Generally the most time-consuming task in aligning range images is searching corresponding points. The fastest searching method is the "Inverse Calibration" method. However, this method requires pre-computed lookup tables and precise sensor parameters. We propose a fast searching method using "index images", which work as look-up tables and are rapidly created without any sensor parameters by using graphics hardware. To accelerate the computation to estimate rigid transformations, we employed a linear error evaluation method. When the number of range images increases, the computation time for solving the linear equations becomes too long because of the large size of the coefficient matrix. On the other hand, the coefficient matrix has the characteristic of becoming sparser as the number of range images increases. Thus, we applied the Incomplete Cholesky Conjugate Gradient (ICCG) method to solve the equations and found that the ICCG greatly accelerates the matrix operation by pre-conditioning the coefficient matrix. Some experimental results in which a large number of range images are aligned demonstrate the effectiveness of our method. © 2008 Springer-Verlag US.

DOI： 10.1007/978-0-387-75807_6

Scopus

researchmap

Language：English   Publishing type：Research paper (scientific journal)   Publisher：TAYLOR & FRANCIS LTD  

1. Inter-breed morphological comparisons were made among 11 breeds of Japanese native chickens (Gifujidori, Hinaidori, Shokoku, Totenko, Tomaru, Satsumadori, Shamo, Koshamo, Koeyoshi, Chabo and Nagoya), White Leghorn, broiler chickens (Chunky) and red junglefowl collected in the Philippines, based on results of direct measurements and analysis by computer vision techniques of the skull.
2. Analysis of direct measurements identified two groups of chicken: a small type that included the Chabo, Koshamo, red junglefowl, Gifujidori and Shokoku and a large type that included the remaining breeds studied. These groupings were made based on size determined both in the first (PC1) and second principal component (PC2). The greatest length of the cranium and condylobasal length greatly contributed to the morphological differences between these two groups.
3. Analysis by computer vision techniques, however, identified three groups: the Bantam group (which includes red junglefowl), Shokoku group and Shamo group. White Leghorn clustered within the Shokoku group while the broiler chicken belonged to the Shamo group. The region around the junction of the neural cranium and the visceral cranium contributed greatly to the morphological differences among breeds, both in the PC1 and PC2.

DOI： 10.1080/00071660802094727

Web of Science

J-GLOBAL

researchmap

Language：English   Publishing type：Part of collection (book)   Publisher：Springer US  

This chapter describes a method for parallel alignment of multiple range images. There are problems of computational time and memory space in aligning a large number of range images simultaneously. We developed a parallel method to address the problems. Searching for corresponding points between two range images is time-consuming and requires considerable memory space when performed independently. However, this process can be preformed in parallel, with each corresponding pair of range images assigned to a node. Because the computation time is approximately proportional to the number of vertices, by assigning the pairs so that the number of vertices computed is equal on each node, the load on each node is effectively distributed. In order to reduce the amount of memory required on each node, a hypergraph that represents the correspondences of range images is created, and heuristic graph partitioning algorithms are applied to determine the optimal assignment of the pairs. Moreover, by rejecting redundant dependencies, it becomes possible to accelerate computation time and reduce the amount of memory required on each node. The method was tested on a 16-processor PC cluster, where it demonstrated high extendibility and improved performance. © 2008 Springer-Verlag US.

DOI： 10.1007/978-0-387-75807_7

Scopus

researchmap

Language：English   Publishing type：Research paper (scientific journal)   Publisher：SPRINGER  

This paper presents an overview of our research project on digital preservation of cultural heritage objects and digital restoration of the original appearance of these objects. As an example of these objects, this project focuses on the preservation and restoration of the Great Buddhas. These are relatively large objects existing outdoors and providing various technical challenges. Geometric models of the great Buddhas are digitally achieved through a pipeline, consisting of acquiring data, aligning multiple range images, and merging these images. We have developed two alignment algorithms: a rapid simultaneous algorithm, based on graphics hardware, for quick data checking on site, and a parallel alignment algorithm, based on a PC cluster, for precise adjustment at the university. We have also designed a parallel voxel-based merging algorithm for connecting all aligned range images. On the geometric models created, we aligned texture images acquired from color cameras. We also developed two texture mapping methods. In an attempt to restore the original appearance of historical objects, we have synthesized several buildings and statues using scanned data and a literature survey with advice from experts.

DOI： 10.1007/s11263-007-0039-y

Web of Science

researchmap

Language：English   Publishing type：Research paper (scientific journal)   Publisher：SAGE PUBLICATIONS LTD  

This paper proposes a framework that achieves the Learning from Observation paradigm for learning dance motions. The framework enables a humanoid robot to imitate dance motions captured from human demonstrations. This study especially focuses on leg motions to achieve a novel attempt in which a biped-ope robot imitates not only upper body motions but also leg motions including steps. Body differences between the robot and the original dancer make the problem difficult because the differences prevent the robot front straight forward-v following the original motions and they also change dynamic body balance. We propose leg task models, which play a key role in solving the problem. Low-level tasks in leg motion are modelled so that they clearly provide essential information required,for keeping dynamic stability and important motion characteristics. The models divide the problem of adapting motions into the problem of recognizing a sequence of the tasks and the problem of executing the task sequence. We have developed a method for recognizing the tasks from captured motion data and a method for generating the motions of the tasks that can be executed by existing robots including HRP-2. HRP-2 successfully performed the generated motions, which imitated a traditional folk dance performed by human dancers.

DOI： 10.1177/0278364907079430

Web of Science

researchmap

Language：English   Publishing type：Research paper (scientific journal)   Publisher：IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG  

Reconstruction of real-world scenes from a set of multiple images is a topic in computer vision and 3D computer graphics with many interesting applications. Attempts have been made to real-time reconstruction on PC cluster systems. While these provide enough performance, they are expensive and less flexible. Approaches that use a GPU hardware-acceleration on single workstations achieve real-time framerates for novel-view synthesis, but do not provide an explicit volumetric representation. This work shows our efforts in developing a GPU hardware-accelerated framework for providing a photo-consistent reconstruction of a dynamic 3D scene. High performance is achieved by employing a shape from silhouette technique in advance. Since the entire processing is done on a single PC, the framework can be applied in mobile environments, enabling a wide range of further applications. We explain our approach using programmable vertex and fragment processors and compare it to highly optimized CPU implementations. We show that the new approach can outperform the latter by more than one magnitude and give an outlook for interesting future enhancements.

DOI： 10.1093/ietisy/e90-d.8.1175

Web of Science

researchmap

Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：SPRINGER-VERLAG BERLIN  

This paper proposes a novel volume-based motion capture method using a bottom-up analysis of volume data and an example topology database of the human body. By using a two-step graph matching algorithm with many example topological graphs corresponding to postures that a human body can take, the proposed method does not require any initial parameters or iterative convergence processes, and it can solve the changing topology problem of the human body. First, three-dimensional curved lines (skeleton) are extracted from the captured volume data using the thinning process. The skeleton is then converted into an attributed graph. By using a graph matching algorithm with a large amount of example data, we can identify the body parts from each curved line in the skeleton. The proposed method is evaluated using several video sequences of a single person and multiple people, and we can confirm the validity of our approach.

Web of Science

researchmap

Language：Japanese   Publishing type：Research paper (other academic)  

researchmap

Language：Japanese   Publishing type：Research paper (other academic)  

researchmap

Language：Japanese   Publisher：The Robotics Society of Japan  

In this paper, we propose a method that enables a biped humanoid robot to reproduce human dance motions with its whole body. Our method is based on the paradigm ofLearning from Observation. In this study, a robot uses its own legs to support the body during a dance performance. We proposeleg task models, which can solve the problems caused by severe constraints in adapting human motions to the legs of a robot. First, elements of the leg task models are recognized from motion data captured from human performances. Then motion data of a robot is regenerated from the recognized elements so that the motion is stably executable on the robot. Our method was verified by experiments on a humanoid robotHRP-2using a traditional folk dance. HRP-2 successfully performed dance motions that were automatically reproduced from motion data captured from human dance performances.

DOI： 10.7210/jrsj.24.388

CiNii Article

CiNii Books

researchmap

Other Link： https://jlc.jst.go.jp/DN/JALC/00278397673?from=CiNii

Language：Japanese   Publishing type：Research paper (other academic)  

researchmap

Language：Japanese   Publishing type：Research paper (other academic)  

researchmap

Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：SPRINGER-VERLAG BERLIN  

We propose a new rendering system for large-scale, 3D geometic data that can be used with web-based content management systems (CMS). To achieve this, we employed a geometry hierarchical encoding method "QSplat" and implemented this in a Java and JOGL (Java bindings of OpenGL) environment. Users can view large-scale geometric data using conventional HTML browsers with a non-powerful CPU and low-speed networks. Further, this system is independent of the platforms. We add new functionalities so that users can easily understand the geometric data: Annotations and HTML Synchronization. Users can see the geometric data with the associated annotations that describe the names or the detailed explanations of the particular portions. The HTML Synchronization enables users to smoothly and interactively switch our rendering system and HTML contents. The experimental results show that our system performs an interactive frame rate even for a large-scale data whereas other systems cannot render them.

DOI： 10.1007/11941354_126

Web of Science

researchmap

Language：English   Publishing type：Research paper (scientific journal)   Publisher：WILEY-BLACKWELL  

In computer graphics, considerable research has been conducted on realistic human motion synthesis. However most research does not consider human emotional aspects, which often strongly affect human motion. This paper presents a new approach for synthesizing dance performance matched to input music, based on the emotional aspects of dance performance. Our method consists of a motion analysis, a music analysis, and a motion synthesis based on the extracted features. In the analysis steps, motion and music feature vectors are acquired. Motion vectors are derived from motion rhythm and intensity, while music vectors are derived from musical rhythm, structure, and intensity. For synthesizing dance performance, we first find candidate motion segments whose rhythm features are matched to those of each music segment, and then we find the motion segment set whose intensity is similar to that of music segments. Additionally, our system supports having animators control the synthesis process by assigning desired motion segments to the specified music segments. The experimental results indicate that our method actually creates dance performance as if a character was listening and expressively dancing to the music.

DOI： 10.1111/j.1467-8659.2006.00964.x

Web of Science

researchmap

Language：Japanese   Publishing type：Research paper (other academic)   Publisher：The Institute of Electronics, Information and Communication Engineers  

Three dimensional geometry information, as well as live video image, of a disaster site is useful for damage investigation and rescue planning. In this report, presented is an information gathering mobile robot that acquires geometry information of a disaster site. After moving to every location directed by a remote operator, the robot acquires omnidirectional range data by using an omnidirectional range sensor, and estimates its position and orientation from the amount of wheels' rotation and an orientation sensor. Using the estimated location as the initial value, current and past range data are registered by the ICP algorithm. Indoor experiments have shown that the self-localization error was within five percents.

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publishing type：Research paper (other academic)  

researchmap

Language：Japanese   Publishing type：Research paper (other academic)  

researchmap

Language：Japanese   Publishing type：Research paper (other academic)  

researchmap

Language：Japanese   Publishing type：Research paper (other academic)  

researchmap

Language：English   Publisher：Yamashina Institute for Ornitology  

DOI： 10.3312/jyio.36.120

CiNii Article

researchmap

Language：Japanese  

researchmap

Language：Japanese   Publishing type：Research paper (other academic)  

researchmap

Language：English   Publishing type：Research paper (international conference proceedings)  

researchmap

Language：Japanese   Publishing type：Research paper (other academic)  

researchmap

Language：Japanese   Publishing type：Research paper (other academic)  

researchmap

Language：Japanese   Publishing type：Research paper (other academic)  

researchmap

Language：Japanese   Publishing type：Research paper (other academic)  

researchmap

Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：IEEE  

The goal of the study described in this paper is to develop a total technology for archiving human dance motions. A key feature of this technology is a dance replay by a humanoid robot. Although human dance motions can be acquired by a motion capture system, a robot cannot exactly follow the captured data because of different body structure and physical properties between the human and the robot. In particular, leg motions are too constrained to be converted from the captured data because the legs must interact with the floor and keep dynamic balance within the mechanical constraints of current robots. To solve this problem, we have designed a symbolic description of leg motion primitives in a dance performance. Human dance actions are recognized as a sequence of primitives and the same actions of the robot can be regenerated from them. This framework is more reasonable than modifying the original motion to adapt the robot constraints. We have developed a system to generate feasible robot motions from a human performance, and realized a dance performance by the robot HRP-1S.

Web of Science

researchmap

Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：IEEE Computer Society  

This paper overviews our research on digital preservation of cultural assets and digital restoration of their original appearance. Geometric models are digitally achieved through a pipeline consisting of scanning, registering and merging multiple range images. We have developed a robust simultaneous registration method and an efficient and robust voxel-based integration method. On the geometric models created, we have to align texture images acquired from a color camera. We have developed two texture mapping methods. In an attempt to restore the original appearance of historical heritage objects, we have synthesized several buildings and statues using scanned data and literature survey with advice from experts.

DOI： 10.1109/CVPRW.2003.10001

Scopus

researchmap

Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：IEEE  

The number of Japanese traditional dancers has been decreasing. Without performers, some dances will be disappeared, because they can not be recorded by conventional media, such as paper We have proposed an archiving method specifically for dancing pattern. Our method has four main stages: 1. Digitizing motions by motion capture systems, 2. Analyzing motions, 3. Synthesizing motions, 4. Reproducing the dance motions by CG and humanoid robots. In order to effectively record the moving patterns, motion primitives are extracted. Each motion primitive describes each basic motion. However most previous primitive extraction fails to provide the rhythm which results in unrhythmic synthesized motion. In this paper we propose a new motion analysis method, which integrates music rhythm into the motion primitives. Our experiment confirmed that our motion analysis yielded the motion primitives in accordance to the music rhythm.

Web of Science

researchmap

Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：IEEE  

This paper presents the method for synthesizing stylistic human motions through visual observation. The human motion data is acquired from a motioncapture system. The whole motion sequence is divided into some motion elements and clusterd into some groups according to the correlation of end-effectors' trajectories. We call these segments as 'motion primitives'. Concatenating these motion primitives, we can generates new dance motions. We also think of a motion primitive consists of a basic motion and a motion style. The basic motion is common to all dancers, and the style represents their characteristics. We extracted these two components through the further analysis steps. The experiment results shows the validity of our approach.

Web of Science

researchmap

Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：IEEE  

The human body motion synthesis is highly necessary for humanoid robots' motion planning and computer animations. In this paper, new method for generating human-like natural motions based on the motion database acquired by motion capture systems is described. On the analysis step, the acquired motions are divided into some motion segments, and then the characteristic poses and motions are archived as 'motion styles'. The motion style is a kind of the human skill, and it's unique to the motions' scenario, such as the different kinds of dances. On the synthesis step, users direct the key poses of human figures. The system generates the characteristic motions according to the user's directions and motion style database. The experiment result shows that this method can synthesize the realistic 'stylized' motions with this framework.

Web of Science

researchmap

Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：IEEE COMPUTER SOC  

This paper describes a method for parallel alignment of multiple range images. It is difficult to align a large number of range images simultaneously. Therefore, we developed the parallel method to improve the time and memory performances of the alignment process. Although a general simultaneous alignment algorithm searches correspondences for all pairs of all range images by rejecting redundant dependencies, our method makes it possible to accelerate computation time and reduce the amount of memory used Since the computation between two range images can be preformed independently, each correspondence pair of range images is assigned to each node. Because the computation time is proportional to the number of vertices assigned to each node, by assigning the pairs so that the number of vertices computed is equal on each node, the load on each node is effectively distributed. The heuristic algorithms for graph partitioning are applied to this problem in order to reduce the amount of memory used on each node. The method was tested on a 16 processor PC cluster, where it demonstrated the high extendibility and the performance improvement in time and memory.

DOI： 10.1109/IM.2003.1240250

Web of Science

Scopus

researchmap

Publishing type：Research paper (international conference proceedings)   Publisher：IEEE  

We propose a method which refines the range measurement of range finders by computing correspondences of vertices of multiple range images acquired from various viewpoints. Our method assumes that a range image acquired by a laser range finder has anisotropic error distribution which is parallel to the ray direction. Thus, we find corresponding points of range images along with the ray direction. We iteratively converge range images to minimize the distance of corresponding points. We demonstrate the effectiveness of our method by presenting the experimental results of artificial and real range data. Also we show that our method refines a 3D shape more accurately as opposed to that achieved by using the Gaussian filter.

DOI： 10.1007/978-0-387-75807_11

Scopus

researchmap

Other Link： http://dblp.uni-trier.de/db/conf/iros/iros2002.html#conf/iros/SagawaONKI02

J-GLOBAL

researchmap

J-GLOBAL

researchmap

J-GLOBAL

researchmap

J-GLOBAL

researchmap

J-GLOBAL

researchmap

researchmap

J-GLOBAL

researchmap

researchmap

researchmap

researchmap

J-GLOBAL

researchmap

J-GLOBAL

researchmap

J-GLOBAL

researchmap

J-GLOBAL

researchmap

J-GLOBAL

researchmap

J-GLOBAL

researchmap

Language：Japanese   Publisher：学習院大学計算機センター  

J-GLOBAL

researchmap

J-GLOBAL

researchmap

researchmap

J-GLOBAL

researchmap

Language：Japanese  

J-GLOBAL

researchmap

Language：English   Publishing type：Article, review, commentary, editorial, etc. (international conference proceedings)   Publisher：Springer Verlag  

In this paper, we show the relationship between pupil dilation and visualmanipulation tasks to measure the magnitude of individual habituation effect and task difficulty. Our findings show that pupil dilation can be used as a new physiological signal in the application of audience measurement, affective computing, affective communications, and user interface design. We built a pointer maze game where a subject moves a pointer from start to end positions on a straight pathway, and we observe the subject’s pupil size while changing the pathway width and performing the game repeatedly. Through the two experiments, we found the maximum pupil size increases during the game when the pathway narrows. The first experiment indicates the difficulty of the task (narrower pathway) is related to the larger pupil diameter. On the basis of these results, we built models relating to (1) pupil size and pathway width, (2) pupil size and duration, and (3) pathway width and duration. The second experiment indicates the pupil constriction is related to the habituation effect of the users. While a similar effect has already been reported, the magnitude of pupil dilation during our task was about ten times as high as that in other tasks, so our confidence in the model is high.

DOI： 10.1007/978-3-319-56687-0_3

Scopus

researchmap

Language：English   Publishing type：Article, review, commentary, editorial, etc. (international conference proceedings)  

researchmap

Language：Japanese  

CiNii Article

researchmap

Language：Japanese  

CiNii Article

CiNii Books

researchmap

Language：English   Publishing type：Research paper, summary (international conference)   Publisher：ASSOC COMPUTING MACHINERY  

Visual inspection of medical imagery such as MRI and CT scans is a major task for medical professionals who must diagnose and treat patients without error. Given this goal, visualizing search behavior patterns used to recognize abnormalities in these images is of interest. In this paper we describe the development of a system which automatically generates multiple image-dependent heat maps from eye gaze data of users viewing medical image slices. This system only requires the use of a non-wearable eye gaze tracker and video capturing system. The main automated features are the identification of a medical image slice located inside a video frame and calculation of the correspondence between display screen and raw image eye gaze locations. We propose that the system can be used for eye gaze analysis and diagnostic training in the medical field.

DOI： 10.1145/2857491.2857504

Web of Science

researchmap

Language：English   Publishing type：Article, review, commentary, editorial, etc. (international conference proceedings)   Publisher：IEEE COMPUTER SOC  

Image registration is an important and fundamental problem in computer vision and image processing. Although there are currently a large number of image registration algorithms such as RANSAC and its extensions, image registration under very noisy conditions remains difficult when it cannot obtain enough number of correct corresponding points. This paper solves this issue by introducing a random resample consensus (RANRESAC) strategy, which achieves robust registration where it is difficult to obtain enough numbers of correct correspondence pairs. In contrast to RANSAC, proposed RANRESAC newly generate corresponding points for the images using the hypothesis transformation function, and verifies the correctness by evaluating the similarity of the local features at the newly sampled points. To confirm the effectiveness for the proposed method, we first conducted an preliminary experiment that evaluates the similarity of texture and orientation components of SURF local descriptor in the images adding several levels of noise. As the result, we observed the texture component is more stable than the orientation component. Based on this finding, we design the RANRESAC algorithm and performed experiments using a open image registration dataset. As the result, proposed method outperforms to the RANSAC, MSAC and Optimal RANSAC algorithms in large noise conditions.

DOI： 10.1109/ICPR.2016.7899742

Web of Science

researchmap

J-GLOBAL

researchmap

Language：Japanese   Publishing type：Meeting report  

researchmap

Language：Japanese   Publishing type：Article, review, commentary, editorial, etc. (international conference proceedings)  

researchmap

Language：Japanese  

CiNii Article

CiNii Books

researchmap

Language：English  

researchmap

Language：Japanese  

researchmap

Language：English   Publishing type：Research paper, summary (international conference)   Publisher：IEEE  

We present a mobile human view estimation system using a corneal imaging technique. Compared to the current eye gaze tracking (EGT) systems, our system does not require per-session calibrations and a frontal view (scene) camera, making it suitable for wearable glass systems because it is easier to use and more socially acceptable due to the lack of a frontal scene camera. Our glasses system consists of a glass frame and a micro eye camera that captures the eye (corneal) reflections of a user. 3D corneal pose tracking is performed for the captured images by using a particle filter-based real-time tracking method leveraged by a 3D eye model and weak perspective projection. We then compute the gaze reflection point (GRP) where the light from the point of gaze (PoG) is reflected, enabling us to identify where a user is looking in a scene image reflected on the corneal surface. We conducted experiments using a standard computer display setup and several real-world scenes, and found that the proposed method performs with considerable accuracy under non-calibrated setups. This demonstrates its potential for various purposes such as the user interface of a glasses systems and the analysis of human perceptions in actual scenes for marketing, environmental design, and quality-of-life applications.

Web of Science

researchmap

Language：Japanese  

researchmap

Language：English   Publishing type：Research paper, summary (international conference)   Publisher：IEEE  

We present a mobile human view estimation system using a corneal imaging technique. Compared to the current eye gaze tracking (EGT) systems, our system does not require per-session calibrations and a frontal view (scene) camera, making it suitable for wearable glass systems because it is easier to use and more socially acceptable due to the lack of a frontal scene camera. Our glasses system consists of a glass frame and a micro eye camera that captures the eye (corneal) reflections of a user. 3D corneal pose tracking is performed for the captured images by using a particle filter-based real-time tracking method leveraged by a 3D eye model and weak perspective projection. We then compute the gaze reflection point (GRP) where the light from the point of gaze (PoG) is reflected, enabling us to identify where a user is looking in a scene image reflected on the corneal surface. We conducted experiments using a standard computer display setup and several real-world scenes, and found that the proposed method performs with considerable accuracy under non-calibrated setups. This demonstrates its potential for various purposes such as the user interface of a glasses systems and the analysis of human perceptions in actual scenes for marketing, environmental design, and quality-of-life applications.

Web of Science

researchmap

Language：English  

researchmap

Language：English   Publishing type：Research paper, summary (international conference)  

researchmap

Language：Japanese  

researchmap

Language：Japanese  

researchmap

Language：English  

researchmap

Language：English  

researchmap

Language：Japanese  

researchmap

Language：Japanese  

CiNii Article

CiNii Books

researchmap

Other Link： http://id.ndl.go.jp/bib/025616715

Language：English   Publisher：TUT Press  

This book covers an approach to conversational informatics which encompasses science and technology for understanding and augmenting conversation in the network age. A major challenge in engineering is to develop a technology for conveying not just messages but also underlying wisdom. Relevant theories and practices in cognitive linguistics and communication science, as well as techniques developed in computational linguistics and artificial intelligence, are discussed.

DOI： 10.1007/978-4-431-55040-2_1

Scopus

researchmap

Language：English  

CiNii Article

CiNii Books

researchmap

Language：Japanese  

CiNii Article

CiNii Books

researchmap

J-GLOBAL

researchmap

Language：English   Publisher：IEEE COMPUTER SOC  

Eye-gaze tracking (EGT) is an important problem with a long history and various applications. However, state-of-the-art geometric vision-based techniques still suffer from major limitations, especially (1) the requirement for calibration of a static relationship between eye camera and scene, and (2) a parallax error that occurs when the depth of the scene varies. This paper introduces a novel concept for EGT that overcomes these limitations using corneal imaging. Based on the observation that the cornea reflects the surrounding scene over a wide field of view, it is shown how to extract that information and determine the point of gaze (PoG) directly in an eye image. To realize this, a closed-form solution is developed to obtain the gaze-reflection point (GRP), where light from the PoG reflects at the corneal surface into a camera. This includes compensation for the individual offset between optical and visual axis. Quantitative and qualitative evaluation shows that the strategy achieves considerable accuracy and successfully supports depth-varying environments. The novel approach provides important practical advantages, including reduced intrusiveness and complexity, and support for flexible dynamic setups, non-planar scenes and outdoor application.

DOI： 10.1109/ACPR.2013.84

Web of Science

researchmap

Language：English   Publisher：IEEE  

This paper discusses the usefulness of human body-parts tracking for acquiring subtle cues in social interactions. While many kinds of body-parts tracking algorithms have been proposed, we focus on particle filtering-based tracking using prior models, which have several advantages for researches on social interactions. As a first step for extracting subtle cues from videos of social interaction behaviors, the advantages, disadvantages, and prospective properties of the body-parts tracking using prior models are summarized with actual results.

DOI： 10.1109/ICCVW.2013.106

Web of Science

researchmap

Language：Japanese   Publishing type：Research paper, summary (national, other academic conference)  

DOI： 10.1007/978-3-642-33709-3_12

researchmap

Language：English  

researchmap

Language：English   Publisher：IEEE  

Learning by imitation is becoming increasingly important for teaching humanoid robots new skills. The simplest form of imitation is behavior copying in which the robot is minimizing the difference between its perceived motion and that of the imitated agent. One problem that must be solved even in this simplest of all imitation tasks is calculating the learner's pose corresponding to the perceived pose of the agent it is imitating. This paper presents a general framework for solving this problem in closed form for the arms of a generalized humanoid robot of which most available humanoids are special cases. The paper also reports the evaluation of the proposed system for real and simulated robots.

DOI： 10.1109/ROBIO.2013.6739576

Web of Science

researchmap

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

This paper presents a new approach for synthesizing dance performance matched to input music, based on the emotional aspects of dance performance. Our analysis method extracts motion rhythm and intensity from motion capture data and musical rhythm, structure from musical signals. We extract candidates of motion segment sets whose features are matched to those of music segments. For synthesizing dance performance, we select the motion segment set whose intensity by Kinect and connectivity is matched to that of music segments. As the result, our system adds an interactive component to live dancing performed by virtual characters.

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

The corneal imaging technique enables extraction of scene information from corneal reflections and realizes a large number of applications including environment map reconstruction and estimation of a person's area of view. However, since corneal reflection images are usually low quality and resolution, the outcome of the technique is currently limited. To overcome this issue, we propose a first non-central catadioptric approach to reconstruct high-resolution scene information from a series of lower resolution corneal images through a super-resolution technique. We describe a three-step process, including (1) single image environment map recovery, (2) multiple image registration, and (3) high-resolution image reconstruction. In a number of experiments we show that the proposed strategy successfully recovers high-frequency textures that are lost in the source images, and also works with other non-central catadioptric systems, e.g., involving spherical mirrors.

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

This paper presents a new approach for synthesizing dance performance matched to input music, based on the emotional aspects of dance performance. Since the sensor reads express intensities of users' body movements, the system can synthesize character motions whose intensities are synchronized to those of users. Our analysis method extracts motion rhythm and intensity from motion capture data and musical rhythm, structure from musical signals. With the results of this analysis, we generate a motion graph that can generate character motions matched to the musical rhythm. For synthesizing dance performance, we select the motion segment set whose intensity by sensor data and connectivity is matched to that of music segments. As the results, our system adds an interactive component to live dancing performed by virtual characters.

CiNii Article

CiNii Books

researchmap

Language：Japanese  

CiNii Article

researchmap

Language：Japanese  

CiNii Article

researchmap

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

Intuitive and efficient retrieval of motion capture data is essential for effective use of motion capture databases. In this paper, we describe a system that allows the user to retrieve a particular sequence by performing an approximation of the motion with an instrumented puppet. This interface is intuitive because both adults and children have experience playacting with puppets. We develop a novel similarity metric called dual subspace projection method (DSPM) which works even for relatively large databases. We conducted a user study of the system and subjects could find the desired motion with reasonable accuracy from a database consisting of everyday, exercise, and acrobatic behaviors.

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：Information Processing Society of Japan (IPSJ)  

This paper presents a new approach for synthesizing dance performance matched to input music, based on the emotional aspects of dance performance. Our analysis method extracts motion rhythm and inten- sity from motion capture data and musical rhythm, structure from musical signals. We rst extract candidates of motion segment sets whose features are matched to those of music segments. For synthesizing dance perfor- mance, we select the motion segment set whose intensity by accelerometer and connectivity is matched to that of music segments. We evaluate all synthesizing time and matching value.

CiNii Article

CiNii Books

researchmap

Language：English  

Motion-capture-based character animations are widely used in computer graphics and interactive games.In this paper, we show a novel approach to create dancing character animations that react to input music and an accelerometer manipulated by a user. Since the sensor reads express intensities of users' body movements, the system can synthesize character motions whose intensities are synchronized to those of users. Our system consists of analysis phase and synthesis phase. In the analysis phase, the musical beat and segments are detected from input sound, and motion rhythm and intensities are found from motion capture data. With the results of this analysis, we generate a motion graph that can generate character motions matched to the musical rhythm. In synthesis phase, the system receives the output data from an accelerometer and traverses the motion graph according to the matching result between the sensor data and the motion intensity. As the result, our system adds an interactive component to live dancing performed by virtual characters. © 2012 ACM.

DOI： 10.1145/2393347.2396407

Scopus

researchmap

Language：Japanese   Publisher：[大学ICT推進協議会]  

CiNii Article

researchmap

Language：English   Publisher：B M V A PRESS  

The corneal imaging technique enables extraction of scene information from corneal reflections and realizes a large number of applications including environment map reconstruction and estimation of a person's area of view. However, since corneal reflection images are usually low quality and resolution, the outcome of the technique is currently limited. To overcome this issue, we propose a first non-central catadioptric approach to reconstruct high-resolution scene information from a series of lower resolution corneal images through a super-resolution technique. We describe a three-step process, including (1) single image environment map recovery, (2) multiple image registration, and (3) high-resolution image reconstruction. In a number of experiments we show that the proposed strategy successfully recovers high-frequency textures that are lost in the source images, and also works with other non-central catadioptric systems, e.g., involving spherical mirrors. The obtained information about a person and the environment enables novel applications, e.g., for surveillance systems, personal video, human-computer interaction, and upcoming head-mounted cameras (Google Glass [5]).

DOI： 10.5244/C.26.22

Web of Science

researchmap

Language：Japanese  

CiNii Article

researchmap

Language：Japanese  

CiNii Article

CiNii Books

researchmap

Language：Japanese  

CiNii Article

researchmap

Language：English  

Recently, the geometric relation between a human eye and its image has been formalized to analyze environmental light reflections in the cornea. Proceeding with these efforts, our study proposes a theory of the light transport at the corneal surface including multiple eye poses, develops novel applications, and performs comprehensive experimental evaluation. Based on anthropometric data, a spherical-curvature geometric eye model is developed, and subsequently applied to discuss methods for eye pose estimation from projected circular eye features. The combination of camera and corneal mirror acts as a catadioptric imaging system, for which we describe the back projection to reconstruct the position of a light source, and the forward projection of light from a given source into the image. The theory has several practical applications in scene reconstruction and human-computer interaction, where we discuss the geometric calibration of display-camera setups as one particular problem. We propose a novel approach that eliminates the requirement of special hardware and tedious user interaction by analyzing screen reflections in the cornea. Based on this setting, thorough experimental evaluation shows that simple scene reconstruction results in a large error. We discuss possible reasons and introduce an optimization scheme that achieves feasible results by exploiting geometry constraints within the system. Our study provides sophisticated strategies for analyzing the geometric relation between camera, eye pose, corneal shape, and scene structure within arbitrary dynamic environments. The findings and developments enable novel insights, understanding, and applications in the analysis of human-scene interaction. We believe that this work has implications on several fields and is an important contribution.Recently, the geometric relation between a human eye and its image has been formalized to analyze environmental light reflections in the cornea. Proceeding with these efforts, our study proposes a theory of the light transport at the corneal surface including multiple eye poses, develops novel applications, and performs comprehensive experimental evaluation. Based on anthropometric data, a spherical-curvature geometric eye model is developed, and subsequently applied to discuss methods for eye pose estimation from projected circular eye features. The combination of camera and corneal mirror acts as a catadioptric imaging system, for which we describe the back projection to reconstruct the position of a light source, and the forward projection of light from a given source into the image. The theory has several practical applications in scene reconstruction and human-computer interaction, where we discuss the geometric calibration of display-camera setups as one particular problem. We propose a novel approach that eliminates the requirement of special hardware and tedious user interaction by analyzing screen reflections in the cornea. Based on this setting, thorough experimental evaluation shows that simple scene reconstruction results in a large error. We discuss possible reasons and introduce an optimization scheme that achieves feasible results by exploiting geometry constraints within the system. Our study provides sophisticated strategies for analyzing the geometric relation between camera, eye pose, corneal shape, and scene structure within arbitrary dynamic environments. The findings and developments enable novel insights, understanding, and applications in the analysis of human-scene interaction. We believe that this work has implications on several fields and is an important contribution.

CiNii Article

CiNii Books

researchmap

Language：English   Publisher：IEEE COMPUTER SOC  

Display-camera systems enable a large range of vision applications in everyday environments, but require calibrating the pose of the display with respect to the camera. Recently, a promising approach is introduced that analyzes corneal reflections from face images. While having several benefits, the described implementation requires controlled and unnatural conditions, manual elaboration, and offline processing. This work aims to improve the approach and allow for conditions of practice: First, the novel idea is proposed to encode display correspondences into illumination patterns to enable automatic and robust detection. This comprises the first use of coded illumination with corneal reflection analysis. Second, since previous evaluation revealed a large error for the basic algorithm, this work discusses thorough experimental evaluation of an optimization strategy based on geometry constraints in the scene. Results show that considerable improvement can be achieved under common parameter variation, which successfully verifies the feasibility of the approach.

DOI： 10.1109/ACPR.2011.6166661

Web of Science

researchmap

Language：English  

Intuitive and efficient retrieval of motion capture data is essential for effective use of motion capture databases. In this paper, we describe a system that allows the user to retrieve a particular sequence by performing an approximation of the motion with an instrumented puppet. This interface is intuitive because both adults and children have experience playacting with puppets and toys to express particular behaviors or to tell stories with style and emotion. The puppet has 17 degrees of freedom and can therefore represent a variety of motions. We develop a novel similarity metric between puppet and human motion by computing the reconstruction errors of the puppet motion in the latent space of the human motion and those of the human motion in the latent space of the puppet motion. This metric works even for relatively large databases. We conducted a user study of the system and subjects could find the desired motion with reasonable accuracy from a database consisting of everyday, exercise, and acrobatic behaviors. Copyright © 2011 by the Association for Computing Machinery, Inc.

DOI： 10.1145/2019406.2019427

Scopus

researchmap

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

平面ディスプレイを光源とみなし,光源パターンを様々に変化させた対象の画像をカメラで撮影し形状復元やHCIに利用する,ディスプレイ・カメラシステムの研究が行われている.このような用途ではディスプレイとカメラの幾何校正を行うことが必要であり,従来法では球面ミラー等の付加的なデバイスを必要とした.本論文ではこれに対し,ユーザの眼球の表面反射を利用することで,特殊なデバイスを必要とせず校正を実現する手法を提案する.本手法ではディスプレイでパターンを投影し,その反射した像をカメラで撮影することでディスプレイ座標とカメラ座標の対応関係を得る.次に,この対応関係と眼球モデル,及びディスプレイサイズの情報を用いて最適化を行うことで解を得る.11名の被験者を用いた実験及びディスプレイとカメラを様々な位置関係で配置した状況に対する実験を行い,本手法の有効性及び様々な状況下での性能を明らかにした.

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

This paper presents a method for accurately segmenting and classifying 3D range data into particular object classes. Object classification of input images is necessary for applications including robot navigation and automation, in particular with respect to path planning. To achieve robust object classification, we propose the idea of an object feature which represents a distribution of neighboring points around a target point. In addition, rather than processing raw points, we reconstruct polygons from the point data, introducing connectivity to the points. With these ideas, we can refine the Markov Random Field (MRF) calculation with more relevant information with regards to determining "related points". The algorithm was tested against five outdoor scenes and provided accurate classification even in the presence of many classes of interest.

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

We present a novel technique for calibrating display-camera systems from reflections in the user's eyes. Display-camera systems enable a range of vision applications that need controlled illumination, including 3D object reconstruction, facial modeling and human computer interaction. One important issue, though, is the geometric calibration of the display, which requires additional hardware and tedious user interaction. The proposed approach eliminates this requirement by analyzing patterns that are reflected in the cornea, a mirroring device that naturally exists in any display-camera system. We introduce an optimization strategy that is able to refine eye and spherical mirror calibration results. When applied to the eye, it even outperforms spherical mirror calibration unoptimized. Furthermore, we obtain a robust estimation of eye poses which can be used for eye tracking applications. Despite the difficult working conditions, the calibration results are good and should be sufficient for many applications.

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：日本工業出版  

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

CiNii Article

CiNii Books

researchmap

Language：Japanese  

CiNii Article

CiNii Books

researchmap

Other Link： http://id.nii.ac.jp/1001/00062861/

Language：Japanese  

CiNii Article

CiNii Books

researchmap

Other Link： http://id.nii.ac.jp/1001/00062862/

Language：English  

This paper describes a data-driven approach for generating photorealistic animations of human motion. Each animation sequence follows a user-choreographed path and plays continuously by seamlessly transitioning between different segments of the captured data. To produce these animations, we capitalize on the complementary characteristics of motion capture data and video. We customize our capture system to record motion capture data that are synchronized with our video source. Candidate transition points in video clips are identified using a new similarity metric based on 3-D marker trajectories and their 2-D projections into video. Once the transitions have been identified, a video-based motion graph is constructed. We further exploit hybrid motion and video data to ensure that the transitions are seamless when generating animations. Motion capture marker projections serve as control points for segmentation of layers and nonrigid transformation of regions. This allows warping and blending to generate seamless in-between frames for animation. We show a series of choreographed animations of walks and martial arts scenes as validation of our approach. Copyright © 2009 by the Association for Computing Machinery, Inc.

DOI： 10.1145/1507149.1507182

Scopus

researchmap

Language：English   Publisher：IEEE  

This paper presents a method for accurately segmenting and classifying 3D range data into particular object classes. Object classification of input images is necessary for applications including robot navigation and automation, in particular with respect to path planning. To achieve robust object classification, we propose the idea of an object feature which represents a distribution of neighboring points around a target point. In addition, rather than processing raw points, we reconstruct polygons from the point data, introducing connectivity to the points. With these ideas, we can refine the Markov Random Field (MRF) calculation with more relevant information with regards to determining "related points". The algorithm was tested against five outdoor scenes and provided accurate classification even in the presence of many classes of interest.

DOI： 10.1109/ROBOT.2009.5152539

Web of Science

researchmap

Language：English   Publisher：IEEE  

We present a novel technique for calibrating display-camera systems from reflections in the user's eyes. Display-camera systems enable a range of vision applications that need controlled illumination, including 3D object reconstruction, facial modeling and human computer interaction. One important issue, though, is the geometric calibration of the display, which requires additional hardware and tedious user interaction. The proposed approach eliminates this requirement by analyzing patterns that are reflected in the cornea, a mirroring device that naturally exists in any display-camera system. We introduce an optimization strategy that is able to refine eye and spherical mirror calibration results. When applied to the eye, it even outperforms spherical mirror calibration unoptimized. Furthermore, we obtain a robust estimation of eye poses which can be used for eye tracking applications. Despite the difficult working conditions, the calibration results are good and should be sufficient for many applications.

DOI： 10.1109/ICCV.2009.5459330

Web of Science

researchmap

Language：English   Publisher：IEEE  

This work presents a novel approach to registering multiple range images on top of a Google Maps image. The fundamental concept behind the method is matching completely different types of input with each other using classification as a middleman. Range images and Google Maps images are separated into classes, and the range image is also projected into a 2D top-down template image. The template image can then be matched against the Google Maps image to find its location and orientation on the map, which can be used for registering the range images. An experiment comparing this technique against using GPS to find position and orientation showed that it is effective at automatically constructing a reasonable large-scale 3D model whereas GPS would be completely ineffective.

DOI： 10.1109/ICIP.2009.5413885

Web of Science

researchmap

Language：English   Publisher：IEEE  

We present a novel technique for calibrating display-camera systems from reflections in the user's eyes. Display-camera systems enable a range of vision applications that need controlled illumination, including 3D object reconstruction, facial modeling and human computer interaction. One important issue, though, is the geometric calibration of the display, which requires additional hardware and tedious user interaction. The proposed approach eliminates this requirement by analyzing patterns that are reflected in the cornea, a mirroring device that naturally exists in any display-camera system. By applying this strategy we also obtain a continuous estimation of eye poses which facilitates further applications. We investigate the effect of display size, camera-eye distance and individual eye anatomy experimentally using only off-the-shelf components. Results are promising and show the general feasibility of the approach.

DOI： 10.1109/ICIP.2009.5413852

Web of Science

researchmap

J-GLOBAL

researchmap

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

In this research, we propose a method for taking expansive 3D models of environments using highly-accurate laser range data. Laser range data is taken using distance sensors called laser range sensors, which have grown in use over recent years, including such practical applications like Google's street-view. However, trying to model a large-scale environment like a city requires taking data from various locations around the environment, and this data must be aligned with respect to each other. In our method, we initially decide on a location to model, taking range and map data for it and classifying it into multiple regions. We then find the location and orientation on the map that allows the two results to match, giving the location and orientation of the captured range data.

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：Information Processing Society of Japan (IPSJ)  

In this research, we propose a method for taking expansive 3D models of environments using highly-accurate laser range data. Laser range data is taken using distance sensors called laser range sensors, which have grown in use over recent years, including such practical applications like Google's street-view. However, trying to model a large-scale environment like a city requires taking data from various locations around the environment, and this data must be aligned with respect to each other. In our method, we initially decide on a location to model, taking range and map data for it and classifying it into multiple regions. We then find the location and orientation on the map that allows the two results to match, giving the location and orientation of the captured range data.

CiNii Article

CiNii Books

researchmap

Other Link： http://id.nii.ac.jp/1001/00051745/

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR2008) was held in Ancholage, USA, June 22-28 2006. This is a report on CVPR2008 by nine participants.

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：Information Processing Society of Japan (IPSJ)  

IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR2008) was held in Ancholage, USA, June 22-28 2006. This is a report on CVPR2008 by nine participants.

CiNii Article

CiNii Books

researchmap

Language：English   Publisher：The Institute of Electronics, Information and Communication Engineers  

We present a technique for calibrating display-camera systems from corneal reflections in the user's eyes. Display-camera systems enable a range of vision applications that need controlled illumination including 3D object reconstruction, facial modeling or human computer interaction in everyday environments. An important issue is calibrating the pose of the display with respect to the camera. Such a calibration may be achieved using a planar mirror with attached calibration pattern or a spherical mirror of known size. However, all approaches require additional hardware and user interaction. We propose an automatic way to recover display properties from patterns that are reflected in the cornea, a mirroring device that naturally coexists in any display-camera system. By applying this strategy we also obtain a continuous estimation of eye pose which may be used to calibrate eye tracking systems and generally enhance human-computer interaction.

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：Information Processing Society of Japan (IPSJ)  

Recently, considerable research has been conducted on markerless human-body motion capture using volume data. Most studies have used articulated body models that consist of primitives such as cylinders or ellipsoids. However, the methods that use such models require very good initial parameters. This paper proposes a method to find the human posture parameter from just a single frame without any knowledge of previous frames. We first prepare sample volume data which takes various postures, and then cluster them according to the direction of body links. The volumetric feature vector is acquired for each sample and they are learned via the AdaBoost algorithm. The resulting classifier is used for estimating input volume data, and the matched class can be used as an initial parameter for tracking based methods.

CiNii Article

CiNii Books

researchmap

Other Link： http://id.nii.ac.jp/1001/00051779/

Language：English   Publisher：Information Processing Society of Japan (IPSJ)  

We present a technique for calibrating display-camera systems from corneal reflections in the user's eyes. Display-camera systems enable a range of vision applications that need controlled illumination including 3D object reconstruction, facial modeling or human computer interaction in everyday environments. An important issue is calibrating the pose of the display with respect to the camera. Such a calibration may be achieved using a planar mirror with attached calibration pattern or a spherical mirror of known size. However, all approaches require additional hardware and user interaction. We propose an automatic way to recover display properties from patterns that are reflected in the cornea, a mirroring device that naturally coexists in any display-camera system. By applying this strategy we also obtain a continuous estimation of eye pose which may be used to calibrate eye tracking systems and generally enhance human-computer interaction.

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

Recently, considerable research has been conducted on markerless human-body motion capture using volume data. Most studies have used articulated body models that consist of primitives such as cylinders or ellipsoids. However, the methods that use such models require very good initial parameters. This paper proposes a method to find the human posture parameter from just a single frame without any knowledge of previous frames. We first prepare sample volume data which takes various postures, and then cluster them according to the direction of body links. The volumetric feature vector is acquired for each sample and they are learned via the AdaBoost algorithm. The resulting classifier is used for estimating input volume data, and the matched class can be used as an initial parameter for tracking based methods.

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：Information Processing Society of Japan (IPSJ)  

3D video with an arbitrary viewpoint, generated by 3D reconstruction of images captured by multiple cameras, is currently actively researched as a new type of media. In our research, we propose a method to connect different 3D video clips in an attempt to realize interactive 3D video. Conversion of time sequence 3D mesh shapes into skeleton data enables us to find frames that can connect each clip, determine a correspondence between the two mesh shapes, and generate an interpolated path of vertices. The target of our method is a multiple joint model such as a human body, so we can create a human's loop, and naturally switch to a random action at any time. So we can apply this to the creation of interactive media, for example, an object changes when the watcher "touches" it in the virtual immersive space.

CiNii Article

CiNii Books

researchmap

Other Link： http://id.nii.ac.jp/1001/00051926/

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

3D video with an arbitrary viewpoint, generated by 3D reconstruction of images captured by multiple cameras, is currently actively researched as a new type of media. In our research, we propose a method to connect different 3D video clips in an attempt to realize interactive 3D video. Conversion of time sequence 3D mesh shapes into skeleton data enables us to find frames that can connect each clip, determine a correspondence between the two mesh shapes, and generate an interpolated path of vertices. The target of our method is a multiple joint model such as a human body, so we can create a human's loop, and naturally switch to a random action at any time. So we can apply this to the creation of interactive media, for example, an object changes when the watcher "touches" it in the virtual immersive space.

CiNii Article

CiNii Books

researchmap

Language：English  

Industry widely employs the two-dimensional videoconferencing system as a long distance communication tool, but current limitations such as its tendency to misrepresent eye contact prevent it from becoming more widely adopted. We are exploring the possibility of a three-dimensional videoconferencing system for future interactive streaming of point cloud data, and present the preliminary research results in this paper. We have tested thus far with one sender and one receiver, using pre-recorded data for the sender. The sender, encircled by high-definition cameras, stands and speaks in a room. A cluster of computers reconstructs each frame of the camera images into a 3D point cloud and streams it across a high-speed, low-latency network. On the receiving end, a splat-based Tenderer employs a new algorithm to efficiently resample the points in real-time, maintaining a user-specified frame rate. Parallel hardware projects onto multiple screens while head tracking equipment records the viewer's movements, allowing the receiver to view a stereoscopic 3D representation of the sender from multiple angles. We can combine these visuals with appropriate use of multiple audio channels to forge an unparalleled virtual experience. This next step towards immersive 3D videoconferencing brings us closer to empowering worldwide collaboration between research departments. © 2008 IEEE.

DOI： 10.1109/eScience.2008.42

Scopus

researchmap

Language：Japanese   Publisher：Japan Society for Educational Technology  

This paper describes Course Management Systems (CMSs) from the viewpoint of Open Source and Open Standard in terms of Software Architecture. CMS is a key information infrastructure for supporting teaching and learning in higher educational institutions. The effective use of CMS requires the integration with Student Information Systems (SIS), e-Library and other information infrastructures, along with strategic planning and sustainable development. In this paper, we also introduce our latest activities to develop next-generation CMS since 2004.

DOI： 10.15077/jjet.KJ00004964294

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publishing type：Rapid communication, short report, research note, etc. (scientific journal)  

researchmap

Language：English   Publisher：The Institute of Electronics, Information and Communication Engineers  

Reconstruction of real-world scenes from a set of multiple images is a topic in computer vision and 3D computer graphics with many interesting applications. Attempts have been made to real-time reconstruction on PC cluster systems. While these provide enough performance, they are expensive and less flexible. Approaches that use a GPU hardware-acceleration on single workstations achieve real-time framerates for novel-view synthesis, but do not provide an explicit volumetric representation. This work shows our efforts in developing a GPU hardware-accelerated framework for providing a photo-consistent reconstruction of a dynamic 3D scene. High performance is achieved by employing a shape from silhouette technique in advance. Since the entire processing is done on a single PC, the framework can be applied in mobile environments, enabling a wide range of further applications. We explain our approach using programmable vertex and fragment processors and compare it to highly optimized CPU implementations. We show that the new approach can outperform the latter by more than one magnitude and give an outlook for interesting future enhancements.

DOI： 10.1093/ietisy/e90-d.8.1175

CiNii Article

researchmap

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：Information Processing Society of Japan (IPSJ)  

3D human model reconstruction using visual cone intersection between images captured by multiple cameras has been thoroughly researched, but the accuracy of the shape estimation by previous methods is limited due to the use of silhouette images. Increasing the number of cameras can help improve accuracy, but the problems of unsmooth surfaces appearing in the shape data and the disappearance of the surface normal will still remain. In this research, we try to refine this process by reshaping the reconstructed model. First, we estimate a subject's pose from captured human shape data and create a parts-fitted model using standard human model parts prepared in advance. Afterwards, we reshape the parts-fitted model based on the surface normal vector. By doing this, we can obtain an accurate and refined 3D human model that maintains the subject's pose and characteristics of his body.

CiNii Article

CiNii Books

researchmap

Other Link： http://id.nii.ac.jp/1001/00052069/

Language：English   Publisher：IEEE  

This paper introduces a Multi Factor Tensor(MFT) model to recognize motion styles and person identities in dance sequences. We apply a musical information analysis method in segmenting the motion sequence relevant to the key poses and the musical rhythm. We define a task model considering the repeated motion segments, where the motion is decomposed into person invariant factor task and person dependant factor style. We capture the motion data of different people for a few cycles, segment it using the musical analysis approach, normalize the segments using a vectorization method, and realize our MYT model. The experiments are conducted according to two approaches. Various experiments that we conduct to evaluate the potential of the recognition ability of our proposed approaches and the results demonstrate the high accuracy of our model. The recognition results and the motion decomposition with be used in further extending the motion generation process in various styles and for different tasks.

DOI： 10.1109/IROS.2007.4399293

Web of Science

researchmap

Language：Japanese   Publisher：画像電子学会  

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：Information Processing Society of Japan (IPSJ)  

Time sequence 3D data has a broad range of application and is a research area expected for the future development. The time sequence 3D data has such a huge data size that data compression is essential when it is transmitted and saved. This paper proposes a compression method by using the time sequence block-matching with extended 3D Hu invariant moment for the purpose of reduction of the data time redundancy. In this method, the data compression ratio is improved by detecting the rotational motion of an object that is not considered in the existing method.

CiNii Article

CiNii Books

researchmap

Other Link： http://id.nii.ac.jp/1001/00052187/

Language：Japanese   Publisher：Information Processing Society of Japan (IPSJ)  

In this paper, we propose a novel marker free motion capture method by using volume data. A volume data is reconstructed from multiple camera views at each frame through visual hull based method. Then, the thinness process is performed to idenfity the structure of the volume. Here, we can get the model skeleton graph in which a body and limbs are expressed as the nodes, and links expresses the connectivity between them. We compare the acquired graph and the graphs in the Model Graph Database (MGDB) and find the most similar one. The MGDB contains the example graphs which express the human body postures. Because the nodes of the MGDB graph are labeled according to the body portions, we can know the portions of input graph (skeleton) from the graph matching result. Finally we fit the skeleton and human body portions' model by using the identification results of the body portions. The experiment result shows the validity our approach.

CiNii Article

CiNii Books

researchmap

Other Link： http://id.nii.ac.jp/1001/00052185/

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：日本工業出版  

CiNii Article

CiNii Books

researchmap

Language：English  

In this paper we present a novel approach to recognizing motion styles and identifying persons using the Multi Factor Tensor (MFT) model. We apply a musical information analysis method in segmenting the motion sequence relevant to the key poses and the musical rhythm. We define a task model considering the repetitive motion segment, in which motion is decomposed into task and style. Given the motion data set, we formulate the MFT model and factorize it efficiently in recognizing the tasks, the styles, and the identities of persons. In our experiments, traditional dance by several people is chosen as the motion sequence. We capture the motion data for a few cycles, segment it using the musical analysis approach, normalize the segments, and recognize a task model from them. Various experiments to evaluate the potential of the recognition ability of our proposed approach are performed, and the results demonstrate the high accuracy of our model. © 2006 IEEE.

DOI： 10.1109/MFI.2006.265645

Scopus

researchmap

Language：Japanese   Publisher：日本バイオメカニクス学会  

CiNii Article

CiNii Books

researchmap

Language：English   Publisher：IEEE  

This paper proposes a method for synthesizing dance performance synchronized to played music and our method presents a system that imitates dancers' skills in performing their motion while they listen to the music. Our method consists of a motion analysis, a music analysis, and a motion synthesis based on results of the analyses. In these analysis steps, motion and music features are acquired. These features are derived from motion keyframes, motion intensity, music intensity, musical beats, and chord changes. Our system also constructs a motion graph to search similar poses from given dance sequences and to connect them as possible transitions. In the synthesis step, the trajectory that provides the best correlation between music and motion features is selected from the motion graph, and the resulting motion is generated. Our experimental results indicate that our proposed method actually creates dance as the system "hears" the music.

DOI： 10.1109/ROBOT.2006.1642260

Web of Science

researchmap

Language：Japanese   Publisher：一般社団法人情報処理学会  

This paper describes a method for parallel alignment of multiple range images. Since it is difficult to align a large number of range images simultaneously, we developed a parallel method to accelerate and reduce the memory requirement of the process. Although a general simultaneous alignment algorithm searches correspondences for all pairs of all range images, by rejecting redundant dependencies, our method makes it possible to accelerate computation time and reduce the amount of memory used on each node. The correspondence search is performed independently for each pair of range images. A...

CiNii Article

researchmap

Language：Japanese   Publisher：Information Processing Society of Japan (IPSJ)  

This paper describes a method for parallel alignment of multiple range images. Since it is difficult to align a large number of range images simultaneously, we developed a parallel method to accelerate and reduce the memory requirement of the process. Although a general simultaneous alignment algorithm searches correspondences for all pairs of all range images, by rejecting redundant dependencies, our method makes it possible to accelerate computation time and reduce the amount of memory used on each node. The correspondence search is performed independently for each pair of range images. Accordingly, the computations between the pairs are preformed in parallel on multiple processors. All relations between range images are described as a pair node hyper-graph. Then, an optimal pair assignment is computed by partitioning the graph properly. The method was tested on a 16 processor PC cluster, where it demonstrated the high extendibility and the performance improvement in time and memory.

CiNii Article

CiNii Books

researchmap

Other Link： http://id.nii.ac.jp/1001/00010541/

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

In this paper, we present a novel method to estimate several natural quadric (sphere, cylinder, and cone) and plane surface parameters from a range image. First, model surface parameters are estimated using Hough transform technique. Based on these estimated parameters, more reliable parameters are calculated using the EM algorithm. We implemented this method and applied to actual range data.

CiNii Article

CiNii Books

researchmap

CiNii Article

researchmap

Language：English   Publisher：IEEE COMPUTER SOC  

This paper describes a fast and easy-to-use simultaneous alignment method of multiple range images. The most time consuming part of alignment process is searching corresponding points. Although "Inverse calibration" method quickly searches corresponding points in complexity O(n), where n is the number of vertices, the method requires some look-up tables or precise sensor parameters. Then, we propose an easy-to-use method that uses "Index Image": "Index image" can be rapidly created using graphics hardware without precise sensor parameters. For fast computation of rigid transformation matrices of a large number of range images, we utilized linearized error function and applied incomplete Cholesky conjugate gradient (ICCG) method for solving linear equations. Some experimental results that aligned a large number of range images measured with laser range sensors show the effectiveness of our method.

DOI： 10.1109/3DIM.2005.41

Web of Science

researchmap

Language：English   Publisher：IEEE  

The goal of this study is developing a biped humanoid robot that can observe a human dance performance and imitate it. To achieve this goal, we propose a task model of lower body motion, which consists of task primitives (what to do) and skill parameters (how to do it). Based on this model, a sequence of task primitives and their skill parameters are detected from human motion, and robot motion is regenerated from the detected result under constraints of a robot. This model can generate human-like lower body motion including various waist motions as well as various stepping motions of the legs. Generated motions can be performed stably oil an actual robot supported by its own legs. We used improved robot hardware HRP-2, which has superior features in body weight, actuators, and DOF of the waist. By using the proposed method and HRP-2, we have realized a dance performance of Japanese folk dance by the robot, which is synchronized with a performance of a human grand master on the same stage.

DOI： 10.1109/IROS.2005.1545395

Web of Science

researchmap

Language：Japanese  

researchmap

J-GLOBAL

researchmap

Language：Japanese   Publisher：名古屋大学  

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：Information Processing Society of Japan (IPSJ)  

The interaction of robots is the reaction to their outer world according to the outer signals made by humans or environments. In particular, the humanoids which do the cooperative tasks with humans have to be considered how smoothly and naturally communicate with humans. This is achieved by the technologies of the natural motion synthesis or understanding of outer signals. In this paper, we introduce our project which aims to develop the methods to import human motion into humanoids, in particular, human dance motion. Through our research, we concentrated that how to express the 'naturality' of human and synchronization method to the important signals of dances: music signals.

CiNii Article

CiNii Books

researchmap

Other Link： http://id.nii.ac.jp/1001/00052434/

Language：Japanese  

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：Information Processing Society of Japan (IPSJ)  

To realistically model indoor scene is crucial and challenging problem. Indoor scene has a large number of three-dimensional range data and spatial restriction in two-dimensional photographing, as well as inevitable specularities in certain parts. In this paper we intend to deal with such a problem. Principally, our method use three-dimensional (3D) geometrical data and ordinary two-dimensional (2D) color images. The 3D geometrical data is created using parallel alignment and merging of a large number of range data. We texture the 3D data with 2D color images using simultaneous registering technique. Then, we estimate the reflection parameters of the diffuse and specular reflection components from a single image where specularities occur. Finally, using all acquired and estimated data, we can generate synthetic images of real objects realistically.

CiNii Article

CiNii Books

researchmap

Other Link： http://id.nii.ac.jp/1001/00052531/

J-GLOBAL

researchmap

Language：English  

researchmap

Language：English   Publisher：IEEE COMPUTER SOC  

In these days, many important intangible culturalproperties of the world are being lost because of the lack of successive performers. Digital archiving technology is one of the effective solutions for this issue, and we have started our digital archiving project of cultural properties including these intangible ones. For these human motion archives, the method of automatic motion structure analysis is vital for a variety of purposes. We believe that the dance motion consists of "primitive motions" and that motion analysis is necessary to detect these components. Particularly for dance motions, we think these primitives must be synchronized to the musical rhythm. In this paper, we introduce musical information for motion structure analysis. This method automatically detects the musical rhythm and segments the original motion, and classifies them as to the primitive motions. The experimental results confirm that our motion analysis yielded the primitive motions in accordance to the musical rhythm.

DOI： 10.1109/AFGR.2004.1301641

Web of Science

researchmap

Language：English   Publisher：IEEE COMPUTER SOC  

This paper overviews our research on digital preservation of cultural assets and digital restoration of their original appearance. Geometric models are digitally achieved through a pipeline consisting of scanning, registering and merging multiple range images. We have developed a robust simultaneous registration method and an efficient and robust voxel-based integration method. On the geometric models created, we have to align texture images acquired from a color camera. We have developed a texture mapping method to utilize laser reflectance. In an attempt to restore the original appearance of historical heritage objects, we have synthesized several buildings and statues using scanned data and literature survey with advice from experts.

Web of Science

researchmap

Language：Japanese   Publisher：The Institute of Electronics, Information and Communication Engineers  

We have developed a system for humanoid robots to imitate whole body motions performed by a human. Our target motions are traditional folk dances. Human motions can be acquired by a motion capturing system. However, those motion data cannot be imported directly into a robot which have body properties different from that of humans. In order to realize the motion which satisfies the essence of the original dance motion under the robot constraints, we propose a motion recognition and generation method using symbolic motion descriptions. This method have realized an actual performance of "Tsugaru Jongara-Bushi", which is one of the Japanese traditional folk dances, on the humanoid robot "HRP-1S".

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：Information Processing Society of Japan (IPSJ)  

In this paper we propose a new method for estimating a position and color of a light source, as well as reflectance properties of a real object's surface, from a single image. We use the intensity of the diffuse and specular component for estimating the light source position, while the color distribution of the specular region is for estimating the light source color. The flow of this method is basically as follows: first, an initial position of the light source is estimated from a peak location of the specular region and a rough intensity value of the diffuse region. This diffuse-to-specular intensity value is also used to determine the initial values of the object reflectance properties. After having the initial values, using iterative fitting method, the light position and reflectance properties are estimated simultaneously. Finally, the estimation process of the light source color is based on the color distribution of the specular region. By knowing the light source position, color and the object reflectance properties, we can freely generate synthetic images under arbitrary light source conditions.

CiNii Article

CiNii Books

researchmap

Other Link： http://id.nii.ac.jp/1001/00018076/

Language：Japanese   Publisher：Information Processing Society of Japan (IPSJ)  

This paper presents, the method to analyze human motion for the purpose of digital archive of intangible cultural heritages, such as folk dances. In the recent studies, the whole motion sequence is segmented and directly use for computer animation or motion analysis. We proposes the idea that the human dance motion consists of 'Basic Motion' and 'Motion Styles'. The Basic Motion is common motion for any dancers, and Motion Styles represents the uniqueness of the individual dancers. In the experiment, we confirmed proposed method works effectively through different motion data of male and femail dancers.

CiNii Article

CiNii Books

researchmap

Other Link： http://id.nii.ac.jp/1001/00052645/

Language：Japanese   Publisher：Information Processing Society of Japan (IPSJ)  

This paper describes a parallel alignment method of multiple range images. It is difficult to align a large number of range images simultaneously. Therefore, we developed the parallel method to improve the time performance ana memory performance. The method was tested on 16 processor PC cluster and shown the high extendibility and the performance improvement in time and memory.

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：Information Processing Society of Japan (IPSJ)  

In this paper, we consider the geometric and photometric modeling of large-scale and intricately shaped objects, such as cultural heritage objects. When modeling such objects, new issues occurred during the modeling steps which had not been considered in the previous research conducted on the modeling of small, indoor objects. First, when modeling a large-scale and intricately shaped object, a huge amount of data is required to model the object. We would like to propose two approaches to handling this amount of data. We propose a novel method for searching the nearest neighbor by extending the search algorithm using a k-d tree. We constructed a merged model in adaptive resolution according to the geometric and photometric attributes of range images for efficient use of computational resources. Second, we reconstructed a 3D model with an appearance which successfully discarded outliers due to specular reflection, by taking a consensus of the appearance changes of the target object from multiple range images. The photometric attribute of the model can be used for aligning with 2D color images. The third issue is complementing unobservable surfaces of an object. We propose a novel method to complement such holes or gaps in the surfaces, areas which are not observed by any scans. In this novel method, the surface of an object is scanned using various kinds of sensors. We efficiently complement them by taking a consensus of the signed distances of neighbor voxels.

CiNii Article

CiNii Books

researchmap

Language：Japanese   Publisher：Information Processing Society of Japan (IPSJ)  

This study attempts to realize a humanoid robot which can imitate human dances. Human motion can be acquired by a motion capturing system. However, the captured data cannot be directly imported to a robot because of difference of body structure between a human and the robot. Since leg motion must consider balance keeping, constraints of it is too restricted to apply the captured data. This paper proposes a method to recognize human motion and generate robot motion from the recognition result. Primitive motions which is required to express leg motion in dances are defined. A sequence of primitives is extracted from captured motion data. Leg motion of a robot is generated from the primitive sequence to satisfy the leg constraints and the dynamic balance. Generated motion is tested on OpenHRP dynamics simulator. We certified validity of this method.

CiNii Article

CiNii Books

researchmap

Other Link： http://id.nii.ac.jp/1001/00052644/

Language：English   Publisher：IEEE  

The goal of this study is a system for a robot to imitate human dances. This paper describes the process to generate whole body motions which can be performed by an actual biped humanoid robot. Human dance motions are acquired through a motion capturing system. We then extract symbolic representation which is made up of primitive motions: essential postures in arm motions and step primitives in leg motions. A joint angle sequence of the robot is generated according to these primitive motions. Then joint angles are modified to satisfy mechanical constraints of the robot. For balance control, the waist trajectory is moved to acquire dynamics consistency based on desired ZMP. The generated motion is tested on OpenHRP dynamics simulator. In our test, the Japanese folk dance, 'Jongara-bushi' was successfully performed by HPP-1S.

Web of Science

researchmap

Language：English   Publisher：SPIE-INT SOC OPTICAL ENGINEERING  

In this paper, we present an overview of our project to construct a digital archive of cultural heritages. Among the efforts in our project, we briefly overview our research on geometric and photometric preservation of cultural assets and restoration of their original appearance. Digital geometric modeling is achieved through a pipeline consisting of scanning, registering and merging multiple range images. For these purposes, we have developed a robust simultaneous registration method and an efficient and robust voxel based integration method. On top of the geometrical model, we align texture images acquired at the scanning. Because the geometrical relation between the range sensor and the image sensor are calibrated, we automatically align texture images onto the geometrical models. For photometric modeling, we have developed a surface light field based method, which captures the appearance variation of real world objects under different viewpoints and illumination conditions from a series of images. As an attempt to restore the original appearance of historical heritages, we have reconstructed several buildings and statues that have been lost in the past. In this paper, we overview these techniques and show several results of applying the proposed methods to existing ancestral assets.

DOI： 10.1117/12.473086

Web of Science

researchmap

Language：Japanese   Publisher：Forum on Information Technology  

CiNii Article

CiNii Books

researchmap

Language：Japanese  

CiNii Article

CiNii Books

researchmap

CiNii Article

researchmap

Language：English   Publisher：IEEE  

This paper presents the method for importing human dance motion into humanoid robots through visual observation. The human motion data is acquired from a motion capture system consisting of 8 cameras and 8 PC clsters. Then the whole motion sequence is divided into some motion elements and clusterd into some groups according to the correlation of end-effectors' trajectories. We call these segments as 'motion primitives'. New dance motions are generated by concatenating these motion primitives. We are also trying to make a humanoid dance these original or generated motions using inverse-kinematics and dynamic balancing technique.

Web of Science

researchmap

J-GLOBAL

researchmap

Language：Japanese  

CiNii Article

researchmap

Language：Japanese   Publisher：Information Processing Society of Japan (IPSJ)  

This paper proposes the "Distributed Camera System" that can detect human position in a wide area. This system is constructed of many "camera agents" and achieves a task due to cooperation. Camera agents consist of a camera and an image processor and a computer network connects them. They are placed in a real environment and their viewing areas are either overlapping or separated. Each camera agent makes plans using an environmental map and the viewing information of the agent. Using these plans, the system can continuously track a person across all the viewing areas of camera agents. In addition, this system is robust wiht respect to agent's failure. We tested this system in two aspects : the detected trajectories adn the system's robustness. In this paper, we present the results of these evaluations, confirming the efficiency of our system.

CiNii Article

CiNii Books

researchmap

Other Link： http://id.nii.ac.jp/1001/00012173/

CiNii Article

researchmap

CiNii Article

researchmap

Language：English   Presentation type：Oral presentation (invited, special)  

researchmap

Language：English   Presentation type：Oral presentation (invited, special)  

researchmap

Language：Japanese   Presentation type：Public lecture, seminar, tutorial, course, or other speech  

researchmap

Language：English   Presentation type：Oral presentation (general)  

researchmap

Language：Japanese  

researchmap

Language：English  

researchmap

Language：Japanese  

researchmap

Language：Japanese  

researchmap

Language：English  

researchmap

Language：English  

researchmap

Language：Japanese  

researchmap

Language：Japanese  

researchmap

Language：English  

researchmap

Language：English  

researchmap

Language：English  

researchmap

Language：English  

researchmap

Language：English  

researchmap

Language：Japanese  

researchmap

Language：English  

researchmap