Faculty Profiles - HARA Sunao

Papers - HARA Sunao

Division display >> ／ All the affair displays 1 - 62 of about 62

Continual learning on audio scene classification using representative data and memory replay GANs Reviewed International coauthorship International journal

Ibnu Daqiqil ID, Masanobu Abe, Sunao Hara

Bulletin of Electrical Engineering and Informatics 14 ( 1 ) 568 - 580 2025.2

Authorship：Last author Language：English Publishing type：Research paper (scientific journal) Publisher：Institute of Advanced Engineering and Science

This paper proposes a methodology aimed at resolving catastropic forgetting problem by choosing a limited portion of the historical dataset to act as a representative memory. This method harness the capabilities of generative adversarial networks (GANs) to create samples that expand upon the representative memory. The main advantage of this method is that it not only prevents catastrophic forgetting but also improves backward transfer and has a relatively stable and small size. The experimental results show that combining real representative data with artificially generated data from GANs, yielded better outcomes and helped counteract the negative effects of catastrophic forgetting more effectively than solely relying on GAN-generated data. This mixed approach creates a richer training environment, aiding in the retention of previous knowledge. Additionally, when comparing different methods for selecting data as the proportion of GAN-generated data increases, the low probability and mean cluster methods performed the best. These methods exhibit resilience and consistency by selecting more informative samples, thus improving overall performance.

DOI： 10.11591/eei.v14i1.8127

OtologMap: a case study on the construction of an environmental sound map recorded by smart devices at Okayama and Kurashiki Reviewed International journal

Sunao Hara, Masanobu Abe

Noise Control Engineering Journal 1 - 12 2025.2

Explicit Prosody Control to Realize Discourse Focus in End-to-End Text-to-Speech Reviewed International journal

Takumi WADA, Sunao HARA, Masanobu ABE

IEEE International Workshop on Machine Learning for Signal Processing 2024.9

Speech-Emotion Control for Text-to-Speech in Spoken Dialogue Systems Using Voice Conversion and x-vector Embedding Reviewed International journal

Shunichi Kohara, Masanobu Abe, Sunao Hara

2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC 2023) 2280 - 2286 2023.11

Language：English Publishing type：Research paper (international conference proceedings)

In this paper, we propose an algorithm to control both speaker individuality and emotional expressions in synthesized speech, where the most important feature is the controllability of intensity in emotional expressions. An aim of the proposed algorithm is to generate various responses including emotions in text-to-speech (TTS) for spoken dialogue systems (SDS), which results in making the system more human-like. An idea is to control emotion and its intensity in line with the user's utterances. For example, when a user happily talks to SDS, the agent of the SDS responses with happy voice. Generally, voice quality of a user and the agent are different. Therefore, the proposed algorithm consists of two steps: (1) voice conversion to change speaker individuality including emotional expressions and (2) TTS with x-vector acting as an embedding vector to mainly control speech quality related to the intensity of emotions. Evaluation experiments are carried out using a scenario of a spoken dialogue system, where a teacher system of TTS encourages or cheers up students according to students' utterances. The experiment results showed that TTS can successfully reproduce the emotion and its intensity that are extracted from students' utterances, while maintaining the teacher's speaker individuality.

DOI： 10.1109/APSIPAASC58517.2023.10317413

Speech Synthesis Using Ambiguous Inputs From Wearable Keyboards Reviewed International journal

Matsuri Iwasaki, Sunao Hara, Masanobu Abe

2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC 2023) 1172 - 1178 2023.11

Language：English Publishing type：Research paper (international conference proceedings)

This paper proposes a new application in speech communication using text-to-speech (TTS), and the goal is to enable dysarthria, articulation disorder, or persons who have difficulty in speaking to communicate anywhere and anytime using speech to express their thoughts and feelings. To achieve this goal, an input method is required. Thus, we propose a new text-entry method based on three concepts. First, from an easy-to-carry perspective, we used a wearable keyboard that inputs digits from 0 to 9 in decimal notation according to 10-finger movements. Second, from a no-training perspective, users input sentences in a way of touch typing using the wearable keyboard. Following this method, we obtained a sequence of numbers corresponding to the sentence. Third, a neural machine translation (NMT) method is applied to estimate texts from the sequence of numbers. The NMT was trained using two datasets; one is a Japanese-English parallel corpus containing 2.8 million pairs of sentences, which were extracted from TV and movie subtitles, while the other is a Japanese text dataset containing 32 million sentences, which were extracted from a question-and-answer platform. Using the model, phonemes and accent symbols were estimated from a sequence of numbers. Thus, the result accuracy in symbol levels was 91.48% and 43.45% of all the sentences were completely estimated with no errors. To subjectively evaluate feasibility of the NMT model, a two-person word association game was conducted; one gave hints using synthesized speech that is generated from symbols estimated by NMT, while the other guessed answers. As a result, 67.95% of all the quizzes were correctly answered, and experiment results show that the proposed method has the potential for dysarthria to communicate with TTS using a wearable keyboard.

DOI： 10.1109/APSIPAASC58517.2023.10317228

Sound map of urban areas recorded by smart devices: case study at Okayama and Kurashiki Invited International journal

Sunao Hara, Masanobu Abe

Proceedings of the 52nd International Congress and Exposition on Noise Control Engineering (Inter-Noise 2023) 1 - 12 2023.8

Predictions for sound events and soundscape impressions from environmental sound using deep neural networks Invited International journal

Sunao Hara, Masanobu Abe

Proceedings of the 52nd International Congress and Exposition on Noise Control Engineering (Inter-Noise 2023) 1 - 12 2023.8

Prediction method of Soundscape Impressions using Environmental Sounds and Aerial Photographs Reviewed International journal

Yusuke Ono, Sunao Hara, Masanobu Abe

2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) 1222 - 1227 2022.11

Incremental Audio Scene Classifier Using Rehearsal-Based Strategy Reviewed International journal

Ibnu Daqiqil Id, Masanobu Abe, Sunao Hara

2022 IEEE 10th Global Conference on Consumer Electronics (GCCE) 69 - 623 2022.10

Speech-Like Emotional Sound Generation Using WaveNet Reviewed

Kento Matsumoto, Sunao Hara, Masanobu Abe

IEICE Transactions on Information and Systems E105.D ( 9 ) 1581 - 1589 2022.9

Language：English Publishing type：Research paper (scientific journal) Publisher：Institute of Electronics, Information and Communications Engineers (IEICE)

In this paper, we propose a new algorithm to generate Speech-like Emotional Sound (SES). Emotional expressions may be the most important factor in human communication, and speech is one of the most useful means of expressing emotions. Although speech generally conveys both emotional and linguistic information, we have undertaken the challenge of generating sounds that convey emotional information alone. We call the generated sounds "speech-like," because the sounds do not contain any linguistic information. SES can provide another way to generate emotional response in human-computer interaction systems. To generate "speech-like" sound, we propose employing WaveNet as a sound generator conditioned only by emotional IDs. This concept is quite different from the WaveNet Vocoder, which synthesizes speech using spectrum information as an auxiliary feature. The biggest advantage of our approach is that it reduces the amount of emotional speech data necessary for training by focusing on non-linguistic information. The proposed algorithm consists of two steps. In the first step, to generate a variety of spectrum patterns that resemble human speech as closely as possible, WaveNet is trained with auxiliary mel-spectrum parameters and Emotion ID using a large amount of neutral speech. In the second step, to generate emotional expressions, WaveNet is retrained with auxiliary Emotion ID only using a small amount of emotional speech. Experimental results reveal the following: (1) the two-step training is necessary to generate the SES with high quality, and (2) it is important that the training use a large neutral speech database and spectrum information in the first step to improve the emotional expression and naturalness of SES.

DOI： 10.1587/transinf.2021edp7236

Other Link： https://www.webofscience.com/wos/woscc/full-record/WOS:000852731400008

Concept drift adaptation for audio scene classification using high-level features Reviewed International journal

Ibnu Daqiqil Id, Masanobu Abe, Sunao Hara

2022 IEEE International Conference on Consumer Electronics (ICCE) 2022.1

Acoustic Scene Classifier Based on Gaussian Mixture Model in the Concept Drift Situation Reviewed International journal

Ibnu Daqiqil Id, Masanobu Abe, Sunao Hara

Advances in Science, Technology and Engineering Systems Journal 6 ( 5 ) 167 - 176 2021.9

Phonetic and Prosodic Information Estimation from Texts for Genuine Japanese End-to-End Text-to-Speech Reviewed International journal

Naoto Kakegawa, Sunao Hara, Masanobu Abe, Yusuke Ijima

Interspeech 2021 126 - 130 2021.8

Language：English Publishing type：Research paper (international conference proceedings) Publisher：ISCA

The biggest obstacle to develop end-to-end Japanese text-to-speech (TTS) systems is to estimate phonetic and prosodic information (PPI) from Japanese texts. The following are the reasons: (1) the Kanji characters of the Japanese writing system have multiple corresponding pronunciations, (2) there is no separation mark between words, and (3) an accent nucleus must be assigned at appropriate positions. In this paper, we propose to solve the problems by neural machine translation (NMT) on the basis of encoder-decoder models, and compare NMT models of recurrent neural networks and the Transformer architecture. The proposed model handles texts on token (character) basis, although conventional systems handle them on word basis. To ensure the potential of the proposed approach, NMT models are trained using pairs of sentences and their PPIs that are generated by a conventional Japanese TTS system from 5 million sentences. Evaluation experiments were performed using PPIs that are manually annotated for 5,142 sentences. The experimental results showed that the Transformer architecture has the best performance, with 98.0% accuracy for phonetic information estimation and 95.0% accuracy for PPI estimation. Judging from the results, NMT models are promising toward end-to-end Japanese TTS.

DOI： 10.21437/interspeech.2021-914

Model architectures to extrapolate emotional expressions in DNN-based text-to-speech Reviewed International journal

Katsuki Inoue, Sunao Hara, Masanobu Abe, Nobukatsu Hojo, Yusuke Ijima

Speech Communication 126 35 - 43 2021.2

Language：English Publishing type：Research paper (scientific journal) Publisher：Elsevier BV

This paper proposes architectures that facilitate the extrapolation of emotional expressions in deep neural network (DNN)-based text-to-speech (TTS). In this study, the meaning of “extrapolate emotional expressions” is to borrow emotional expressions from others, and the collection of emotional speech uttered by target speakers is unnecessary. Although a DNN has potential power to construct DNN-based TTS with emotional expressions and some DNN-based TTS systems have demonstrated satisfactory performances in the expression of the diversity of human speech, it is necessary and troublesome to collect emotional speech uttered by target speakers. To solve this issue, we propose architectures to separately train the speaker feature and the emotional feature and to synthesize speech with any combined quality of speakers and emotions. The architectures are parallel model (PM), serial model (SM), auxiliary input model (AIM), and hybrid models (PM&AIM and SM&AIM). These models are trained through emotional speech uttered by few speakers and neutral speech uttered by many speakers. Objective evaluations demonstrate that the performances in the open-emotion test provide insufficient information. They make a comparison with those in the closed-emotion test, but each speaker has their own manner of expressing emotion. However, subjective evaluation results indicate that the proposed models could convey emotional information to some extent. Notably, the PM can correctly convey sad and joyful emotions at a rate of 60%.

DOI： 10.1016/j.specom.2020.11.004

Other Link： https://arxiv.org/abs/2102.10345

Module Comparison of Transformer-TTS for Speaker Adaptation based on Fine-tuning Reviewed International journal

Katsuki Inoue, Sunao Hara, Masanobu Abe

Proceedings of APSIPA Annual Summit and Conference (APSIPA-ASC 2020) 826 - 830 2020.12

Language：English Publishing type：Research paper (international conference proceedings) Publisher：IEEE/APSIPA

End-to-end text-to-speech (TTS) models have achieved remarkable results in recent times. However, the model requires a large amount of text and audio data for training. A speaker adaptation method based on fine-tuning has been proposed for constructing a TTS model using small scale data. Although these methods can replicate the target speaker s voice quality, synthesized speech includes the deletion and/or repetition of speech. The goal of speaker adaptation is to change the voice quality to match the target speaker ' s on the premise that adjusting the necessary modules will reduce the amount of data to be fine-tuned. In this paper, we clarify the role of each module in the Transformer-TTS process by not updating it. Specifically, we froze character embedding, encoder, layer predicting stop token, and loss function for estimating sentence ending. The experimental results showed the following: (1) fine-tuning the character embedding did not result in an improvement in the deletion and/or repetition of speech, (2) speech deletion increases if the encoder is not fine-tuned, (3) speech deletion was suppressed when the layer predicting stop token is not fine-tuned, and (4) there are frequent speech repetitions at sentence end when the loss function estimating sentence ending is omitted.

Concept Drift Adaptation for Acoustic Scene Classifier Based on Gaussian Mixture Model Reviewed

Ibnu Daqiqil Id, Masanobu Abe, Sunao Hara

Proceedings of IEEE REGION 10 CONFERENCE (TENCON 2020) 450 - 455 2020.11

Controlling the Strength of Emotions in Speech-Like Emotional Sound Generated by WaveNet Reviewed International journal

Kento Matsumoto, Sunao Hara, Masanobu Abe

Proceedings of Interspeech 2020 3421 - 3425 2020.10

Semi-Supervised Speaker Adaptation for End-to-End Speech Synthesis with Pretrained Models Reviewed International coauthorship International journal

Katsuki Inoue, Sunao Hara, Masanobu Abe, Tomoki Hayashi, Ryuichi Yamamoto, Shinji Watanabe

Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2020) 7634 - 7638 2020.5

Language：English Publishing type：Research paper (international conference proceedings) Publisher：IEEE

Recently, end-to-end text-to-speech (TTS) models have achieved a remarkable performance, however, requiring a large amount of paired text and speech data for training. On the other hand, we can easily collect unpaired dozen minutes of speech recordings for a target speaker without corresponding text data. To make use of such accessible data, the proposed method leverages the recent great success of state-of-the-art end-to-end automatic speech recognition (ASR) systems and obtains corresponding transcriptions from pretrained ASR models. Although these models could only provide text output instead of intermediate linguistic features like phonemes, end-to-end TTS can be well trained with such raw text data directly. Thus, the proposed method can greatly simplify a speaker adaptation pipeline by consistently employing end-to-end ASR/TTS ecosystems. The experimental results show that our proposed method achieved comparable performance to a paired data adaptation method in terms of subjective speaker similarity and objective cepstral distance measures.

DOI： 10.1109/icassp40776.2020.9053371

DNN-based Voice Conversion with Auxiliary Phonemic Information to Improve Intelligibility of Glossectomy Patients' Speech Reviewed International journal

Hiroki Murakami, Sunao Hara, Masanobu Abe

Proceedings of APSIPA Annual Summit and Conference 2019 138 - 142 2019.11

Authorship：Corresponding author Language：English Publishing type：Research paper (international conference proceedings) Publisher：IEEE

In this paper, we propose using phonemic information in addition to acoustic features to improve the intelligibility of speech uttered by patients with articulation disorders caused by a wide glossectomy. Our previous studies showed that voice conversion algorithm improves the quality of glossectomy patients' speech. However, losses in acoustic features of glossectomy patients' speech are so large that the quality of the reconstructed speech is low. To solve this problem, we explored potentials of several additional information to improve speech intelligibility. One of the candidates is phonemic information, more specifically Phoneme Labels as Auxiliary input (PLA). To combine both acoustic features and PLA, we employed a DNN-based algorithm. PLA is represented by a kind of one-of-k vector, i.e., PLA has a weight value (<1.0) that gradually changes in time axis, whereas one-of-k has a binary value (0 or 1). The results showed that the proposed algorithm reduced the mel-frequency cepstral distortion for all phonemes, and almost always improved intelligibility. Notably, the intelligibility was largely improved in phonemes /s/ and /z/, mainly because the tongue is used to sustain constriction to produces these phonemes. This indicates that PLA works well to compensate the lack of a tongue.

DOI： 10.1109/APSIPAASC47483.2019.9023168

Speech-like Emotional Sound Generator by WaveNet Reviewed International journal

Kento Matsumoto, Sunao Hara, Masanobu Abe

Proceedings of APSIPA Annual Summit and Conference 2019 143 - 147 2019.11

Language：English Publishing type：Research paper (international conference proceedings) Publisher：IEEE

In this paper, we propose a new algorithm to generate Speech-like Emotional Sound (SES). Emotional information plays an important role in human communication, and speech is one of the most useful media to express emotions. Although, in general, speech conveys emotional information as well as linguistic information, we have undertaken the challenge to generate sounds that convey emotional information without linguistic information, which results in making conversations in human-machine interactions more natural in some situations by providing non-verbal emotional vocalizations. We call the generated sounds "speech-like", because the sounds do not contain any linguistic information. For the purpose, we propose to employ WaveNet as a sound generator conditioned by only emotional IDs. The idea is quite different from WaveNet Vocoder that synthesizes speech using spectrum information as auxiliary features. The biggest advantage of the idea is to reduce the amount of emotional speech data for the training. The proposed algorithm consists of two steps. In the first step, WaveNet is trained to obtain phonetic features using a large speech database, and in the second step, WaveNet is re-trained using a small amount of emotional speech. Subjective listening evaluations showed that the SES could convey emotional information and was judged to sound like a human voice.

DOI： 10.1109/APSIPAASC47483.2019.9023346

A signal processing perspective on human gait: Decoupling walking oscillations and gestures Reviewed International journal

Adrien Gregorj, Zeynep Yücel, Sunao Hara, Akito Monden, Masahiro Shiomi

Proceedings of the 4th International Conference on Interactive Collaborative Robotics 2019 (ICR 2019) 11659 75 - 85 2019.8

Language：English Publishing type：Research paper (international conference proceedings) Publisher：SPRINGER INTERNATIONAL PUBLISHING AG

This study focuses on gesture recognition in mobile interaction settings, i.e. when the interacting partners are walking. This kind of interaction requires a particular coordination, e.g. by staying in the field of view of the partner, avoiding obstacles without disrupting group composition and sustaining joint attention during motion. In literature, various studies have proven that gestures are in close relation in achieving such goals.Thus, a mobile robot moving in a group with human pedestrians, has to identify such gestures to sustain group coordination. However, decoupling of the inherent -walking- oscillations and gestures, is a big challenge for the robot. To that end, we employ video data recorded in uncontrolled settings and detect arm gestures performed by human-human pedestrian pairs by adopting a signal processing approach. Namely, we exploit the fact that there is an inherent oscillatory motion at the upper limbs arising from the gait, independent of the view angle or distance of the user to the camera. We identify arm gestures as disturbances on these oscillations. In doing that, we use a simple pitch detection method from speech processing and assume data involving a low frequency periodicity to be free of gestures. In testing, we employ a video data set recorded in uncontrolled settings and show that we achieve a detection rate of 0.80.

DOI： 10.1007/978-3-030-26118-4_8

Naturalness Improvement Algorithm for Reconstructed Glossectomy Patient's Speech Using Spectral Differential Modification in Voice Conversion Reviewed International journal

Hiroki Murakami, Sunao Hara, Masanobu Abe, Masaaki Sato, Shogo Minagi

Interspeech 2018, 19th Annual Conference of the International Speech Communication Association 2464 - 2468 2018.9

Language：English Publishing type：Research paper (international conference proceedings) Publisher：ISCA

In this paper, we propose an algorithm to improve the naturalness of the reconstructed glossectomy patient's speech that is generated by voice conversion to enhance the intelligibility of speech uttered by patients with a wide glossectomy. While existing VC algorithms make it possible to improve intelligibility and naturalness, the result is still not satisfying. To solve the continuing problems, we propose to directly modify the speech waveforms using a spectrum differential. The motivation is that glossectomy patients mainly have problems in their vocal tract, not in their vocal cords. The proposed algorithm requires no source parameter extractions for speech synthesis, so there are no errors in source parameter extractions and we are able to make the best use of the original source characteristics. In terms of spectrum conversion, we evaluate with both GMM and DNN. Subjective evaluations show that our algorithm can synthesize more natural speech than the vocoder-based method. Judging from observations of the spectrogram, power in high-frequency bands of fricatives and stops is reconstructed to be similar to that of natural speech.

DOI： 10.21437/Interspeech.2018-1239

Sound sensing using smartphones as a crowdsourcing approach Reviewed International journal

Sunao Hara, Asako Hatakeyama, Shota Kobayashi, Masanobu Abe

2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2017 1328 - 1333 2017.12

Authorship：Lead author,　Corresponding author Language：English Publishing type：Research paper (international conference proceedings) Publisher：IEEE

Sounds are one of the most valuable information sources for human beings from the viewpoint of understanding the environment around them. We have been now investigating the method of detecting and visualizing crowded situations in the city in a sound-sensing manner. For this purpose, we have developed a sound collection system oriented to a crowdsourcing approach and carried out the sound-collection in two Japanese cities, Okayama and Kurashiki. In this paper, we present an overview of sound collections. Then, to show an effectiveness of analyzation by sensed sounds, we profile characteristics of the cities through the visualization results of the sound.

DOI： 10.1109/APSIPA.2017.8282238

An investigation to transplant emotional expressions in DNN-based TTS synthesis Reviewed International journal

Katsuki Inoue, Sunao Hara, Masanobu Abe, Nobukatsu Hojo, Yusuke Ijima

2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2017 1253 - 1258 2017.12

Language：English Publishing type：Research paper (international conference proceedings) Publisher：IEEE

In this paper, we investigate deep neural network (DNN) architectures to transplant emotional expressions to improve the expressiveness of DNN-based text-to-speech (TTS) synthesis. DNN is expected to have potential power in mapping between linguistic information and acoustic features. From multispeaker and/or multi-language perspectives, several types of DNN architecture have been proposed and have shown good performances. We tried to expand the idea to transplant emotion, constructing shared emotion-dependent mappings. The following three types of DNN architecture are examined; (1) the parallel model (PM) with an output layer consisting of both speaker-dependent layers and emotion-dependent layers, (2) the serial model (SM) with an output layer consisting of emotion-dependent layers preceded by speaker-dependent hidden layers, (3) the auxiliary input model (AIM) with an input layer consisting of emotion and speaker IDs as well as linguistics feature vectors. The DNNs were trained using neutral speech uttered by 24 speakers, and sad speech and joyful speech uttered by 3 speakers from those 24 speakers. In terms of unseen emotional synthesis, subjective evaluation tests showed that the PM performs much better than the SM and slightly better than the AIM. In addition, this test showed that the SM is the best of the three models when training data includes emotional speech uttered by the target speaker.

DOI： 10.1109/APSIPA.2017.8282231

New monitoring scheme for persons with dementia through monitoring-area adaptation according to stage of disease Reviewed International journal

Shigeki Kamada, Yuji Matsuo, Sunao Hara, Masanobu Abe

Proceedings of the 1st ACM SIGSPATIAL Workshop on Recommendations for Location-based Services and Social Networks, LocalRec@SIGSPATIAL 2017 1:1 - 1:7 2017.11

Prediction of subjective assessments for a noise map using deep neural networks Reviewed International journal

Shota Kobayashi, Masanobu Abe, Sunao Hara

Adjunct Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers, UbiComp/ISWC 2017 113 - 116 2017.9

Language：English Publishing type：Research paper (international conference proceedings) Publisher：ACM

In this paper, we investigate a method of creating noise maps that take account of human senses. Physical measurements are not enough to design our living environment and we need to know subjective assessments. To predict subjective assessments from loudness values, we propose to use metadata related to where, who and what is recording. The proposed method is implemented using deep neural networks because these can naturally treat a variety of information types. First, we evaluated its performance in predicting five-point subjective loudness levels based on a combination of several features: location specific, participant-specific, and sound-specific features. The proposed method achieved a 16.3 point increase compared with the baseline method. Next, we evaluated its performance based on noise map visualization results. The proposed noise maps were generated from the predicted subjective loudness level. Considering the differences between the two visualizations, the proposed method made fewer errors than the baseline method.

DOI： 10.1145/3123024.3123091

Other Link： http://doi.acm.org/10.1145/3123024.3123091

Speaker Dependent Approach for Enhancing a Glossectomy Patient’s Speech via GMM-Based Voice Conversion

Kei Tanaka, Sunao Hara, Masanobu Abe, Masaaki Sato, Shogo Minagi

Interspeech 2017 2017.8

Speaker Dependent Approach for Enhancing a Glossectomy Patient's Speech via GMM-based Voice Conversion

Kei Tanaka, Sunao Hara, Masanobu Abe, Masaaki Sato, Shogo Minagi

18TH ANNUAL CONFERENCE OF THE INTERNATIONAL SPEECH COMMUNICATION ASSOCIATION (INTERSPEECH 2017), VOLS 1-6 3384 - 3388 2017

Language：English Publishing type：Research paper (international conference proceedings) Publisher：ISCA-INT SPEECH COMMUNICATION ASSOC

In this paper, using GMM-based voice conversion algorithm, we propose to generate speaker-dependent mapping functions to improve the intelligibility of speech uttered by patients with a wide glossectomy. The speaker-dependent approach enables to generate the mapping functions that reconstruct missing spectrum features of speech uttered by a patient without having influences of a speaker's factor. The proposed idea is simple, i.e., to collect speech uttered by a patient before and after the glossectomy, but in practice it is hard to ask patients to utter speech just for developing algorithms. To confirm the performance of the proposed approach, in this paper, in order to simulate glossectomy patients, we fabricated an intraoral appliance which covers lower dental arch and tongue surface to restrain tongue movements. In terms of the Mel-frequency cepstrum (MFC) distance, by applying the voice conversion, the distances were reduced by 25% and 42% for speaker dependent case and speaker-independent case, respectively. In terms of phoneme intelligibility, dictation tests revealed that speech reconstructed by speaker-dependent approach almost always showed better performance than the original speech uttered by simulated patients, while speaker-independent approach did not.

DOI： 10.21437/Interspeech.2017-841

Enhancing a glossectomy patient's speech via GMM-based voice conversion Reviewed International journal

Kei Tanaka, Sunao Hara, Masanobu Abe, Shogo Minagi

2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA) 1 - 4 2016.12

LiBS: Lifelog browsing system to support sharing of memories Reviewed International journal

Atsuya Namba, Sunao Hara, Masanobu Abe

UbiComp 2016 Adjunct - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing 165 - 168 2016.9

Language：English Publishing type：Research paper (international conference proceedings) Publisher：Association for Computing Machinery, Inc

We propose a lifelog browsing system through which users can share memories of their experiences with other users. Most importantly, by using global positioning system data and time stamps, the system simultaneously displays a variety of log information in a time-synchronous manner. This function empowers users with not only an easy interpretation of other users' experiences but also nonverbal notifications. Shared information on this system includes photographs taken by users, Google street views, shops and restaurants on the map, daily weather, and other items relevant to users' interests. In evaluation experiments, users preferred the proposed system to conventional photograph albums and maps for explaining and sharing their experiences. Moreover, through displayed information, the listeners found out their interest items that had not been mentioned by the speakers.

DOI： 10.1145/2968219.2971401

Other Link： http://doi.acm.org/10.1145/2968219.2971401

Safety vs. Privacy: User preferences from the monitored and monitoring sides of a monitoring system Reviewed International journal

Shigeki Kamada, Sunao Hara, Masanobu Abe

UbiComp 2016 Adjunct - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing 101 - 104 2016.9

Language：English Publishing type：Research paper (international conference proceedings) Publisher：Association for Computing Machinery, Inc

In this study, in order to develop a monitoring system that takes into account privacy issues, we investigated user preferences in terms of the monitoring and privacy protec-tion levels. The people on the monitoring side wanted the monitoring system to allow them to monitor in detail. Con-versely, it was observed for the people being monitored that the more detailed the monitoring, the greater the feelings of being surveilled intrusively. Evaluation experiments were performed using the location data of three people in differ-ent living areas. The results of the experiments show that it is possible to control the levels of monitoring and privacy protection without being affected by the shape of a living area by adjusting the quantization level of location informa-tion. Furthermore, it became clear that the granularity of location information satisfying the people on the monitored side and the monitoring side is different.

DOI： 10.1145/2968219.2971412

Other Link： http://doi.acm.org/10.1145/2968219.2971412

Sound collection systems using a crowdsourcing approach to construct sound map based on subjective evaluation Reviewed International journal

Sunao Hara, Shota Kobayashi, Masanobu Abe

IEEE ICME Workshop on Multimedia Mobile Cloud for Smart City Applications (MMCloudCity-2016) 1 - 6 2016

Authorship：Lead author Language：English Publishing type：Research paper (international conference proceedings) Publisher：IEEE

This paper presents a sound collection system that uses crowdsourcing to gather information for visualizing area characteristics. First, we developed a sound collection system to simultaneously collect physical sounds, their statistics, and subjective evaluations. We then conducted a sound collection experiment using the developed system on 14 participants. We collected 693,582 samples of equivalent Aweighted loudness levels and their locations, and 5,935 samples of sounds and their locations. The data also include subjective evaluations by the participants. In addition, we analyzed the changes in sound properties of some areas before and after the opening of a large-scale shopping mall in a city. Next, we implemented visualizations on the server system to attract users' interests. Finally, we published the system, which can receive sounds from any Android smartphone user. The sound data were continuously collected and achieved a specified result.

DOI： 10.1109/ICMEW.2016.7574694

A Spoken Dialog System with Redundant Response to Prevent User Misunderstanding Reviewed International journal

Masaki Yamaoka, Sunao Hara, Masanobu Abe

Proceedings of APSIPA Annual Summit and Conference 2015 229 - 232 2015.12

Language：English Publishing type：Research paper (international conference proceedings) Publisher：IEEE

We propose a spoken dialog strategy for car navigation systems to facilitate safe driving. To drive safely, drivers need to concentrate on their driving; however, their concentration may be disrupted due to disagreement with their spoken dialog system. Therefore, we need to solve the problems of user misunderstandings as well as misunderstanding of spoken dialog systems. For this purpose, we introduced a driver workload level in spoken dialog management in order to prevent user misunderstandings. A key strategy of the dialog management is to make speech redundant if the driver's workload is too high in assuming that the user probably misunderstand the system utterance under such a condition. An experiment was conducted to compare performances of the proposed method and a conventional method using a user simulator. The simulator is developed under the assumption of two types of drivers: an experienced driver model and a novice driver model. Experimental results showed that the proposed strategies achieved better performance than the conventional one for task completion time, task completion rate, and user's positive speech rate. In particular, these performance differences are greater for novice users than for experienced users.

DOI： 10.1109/APSIPA.2015.7415511

Extracting Daily Patterns of Human Activity Using Non-Negative Matrix Factorization Reviewed International journal

Masanobu Abe, Akihiko Hirayama, Sunao Hara

Proceedings of IEEE International Conference on Consumer Electronics (IEEE-ICCE 2015) 36 - 39 2015.1

Language：English Publishing type：Research paper (international conference proceedings) Publisher：IEEE

This paper presents an algorithm to mine basic patterns of human activities on a daily basis using non-negative matrix factorization (NMF). The greatest benefit of the algorithm is that it can elicit patterns from which meanings can be easily interpreted. To confirm its performance, the proposed algorithm was applied to PC logging data collected from three occupations in offices. Daily patterns of software usage were extracted for each occupation. Results show that each occupation uses specific software in its own time period, and uses several types of software in parallel in its own combinations. Experiment results also show that patterns of 144 dimension vectors were compressible to those of 11 dimension vectors without degradation in occupation classification performance. Therefore, the proposed algorithm compressed basic software usage patterns to about one-tenth of their original dimensions while preserving the original information. Moreover, the extracted basic patterns showed reasonable interpretation of daily working patterns in offices.

DOI： 10.1109/ICCE.2015.7066309

Sound collection and visualization system enabled participatory and opportunistic sensing approaches Reviewed International journal

Sunao Hara, Masanobu Abe, Noboru Sonehara

Proceedings of 2015 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops) 390 - 395 2015

Language：English Publishing type：Research paper (international conference proceedings) Publisher：IEEE

This paper presents a sound collection system to visualize environmental sounds that are collected using a crowd sourcing approach. An analysis of physical features is generally used to analyze sound properties; however, human beings not only analyze but also emotionally connect to sounds. If we want to visualize the sounds according to the characteristics of the listener, we need to collect not only the raw sound, but also the subjective feelings associated with them. For this purpose, we developed a sound collection system using a crowdsourcing approach to collect physical sounds, their statistics, and subjective evaluations simultaneously. We then conducted a sound collection experiment using the developed system on ten participants. We collected 6,257 samples of equivalent loudness levels and their locations, and 516 samples of sounds and their locations. Subjective evaluations by the participants are also included in the data. Next, we tried to visualize the sound on a map. The loudness levels are visualized as a color map and the sounds are visualized as icons which indicate the sound type. Finally, we conducted a discrimination experiment on the sound to implement a function of automatic conversion from sounds to appropriate icons. The classifier is trained on the basis of the GMM-UBM (Gaussian Mixture Model and Universal Background Model) method. Experimental results show that the F-measure is 0.52 and the AUC is 0.79.

DOI： 10.1109/PERCOMW.2015.7134069

Other Link： https://ousar.lib.okayama-u.ac.jp/ja/53271

Algorithm to Estimate a Living Area Based on Connectivity of Places with Home Reviewed International journal

Yuji Matsuo, Sunao Hara, Masanobu Abe

HCI International 2015 - Posters’ Extended Abstracts (Part II), CCIS 529 529 570 - 576 2015

Language：English Publishing type：Part of collection (book) Publisher：SPRINGER-VERLAG BERLIN

We propose an algorithm to estimate a person's living area using his/her collected Global Positioning System (GPS) data. The most important feature of the algorithm is the connectivity of places with a home, i.e., a living area must consist of a home, important places, and routes that connect them. This definition is logical because people usually go to a place from home, and there can be several routes to that place. Experimental results show that the proposed algorithm can estimate living area with a precision of 0.82 and recall of 0.86 compared with the grand truth established by users. It is also confirmed that the connectivity of places with a home is necessary to estimate a reasonable living area.

DOI： 10.1007/978-3-319-21383-5_95

Extraction of Key Segments from Day-Long Sound Data Reviewed International journal

Akinori Kasai, Sunao Hara, Masanobu Abe

HCI International 2015 - Posters’ Extended Abstracts (Part I), CCIS 528 528 620 - 626 2015

Language：English Publishing type：Part of collection (book) Publisher：SPRINGER-VERLAG BERLIN

We propose a method to extract particular sound segments from the sound recorded during the course of a day in order to provide sound segments that can be used to facilitate memory. To extract important parts of the sound data, the proposed method utilizes human behavior based on a multisensing approach. To evaluate the performance of the proposed method, we conducted experiments using sound, acceleration, and global positioning system data collected by five participants for approximately two weeks. The experimental results are summarized as follows: (1) various sounds can be extracted by dividing a day into scenes using the acceleration data; (2) sound recorded in unusual places is preferable to sound recorded in usual places; and (3) speech is preferable to nonspeech sound.

DOI： 10.1007/978-3-319-21380-4_105

Sub-Band Text-to-Speech Combining Sample-Based Spectrum with Statistically Generated Spectrum Reviewed International journal

Tadashi Inai, Sunao Hara, Masanobu Abe, Yusuke Ijima, Noboru Miyazaki, Hideyuki Mizuno

Proceedings of Interspeech 2015 264 - 268 2015

Language：English Publishing type：Research paper (international conference proceedings) Publisher：ISCA

As described in this paper, we propose a sub-band speech synthesis approach to develop a high quality Text-to-Speech (TTS) system: a sample-based spectrum is used in the high-frequency band and spectrum generated by HMM-based TTS is used in the low-frequency band. Herein, sample-based spectrum means spectrum selected from a phoneme database such that it is the most similar to spectrum generated by HMM-based speech synthesis. A key idea is to compensate over-smoothing caused by statistical procedures by introducing a sample-based spectrum, especially in the high-frequency band. Listening test results show that the proposed method has better performance than HMM-based speech synthesis in terms of clarity. It is at the same level as HMM-based speech synthesis in terms of smoothness. In addition, preference test results among the proposed method, HMM-based speech synthesis, and waveform speech synthesis using 80 min speech data reveal that the proposed method is the most liked.

Other Link： http://dblp.uni-trier.de/db/conf/interspeech/interspeech2015.html#conf/interspeech/InaiHAIMM15

Inhibitory Effects of an Orally Active Small Molecule Alpha4beta1/Alpha4beta7 Integrin Antagonist, TRK-170, on Spontaneous Colitis in HLA-B27 Transgenic Rats

Hiroe Hirokawa, Yoko Koga, Rie Sasaki, Sunao Hara, Hiroyuki Meguro, Mie Kainoh

GASTROENTEROLOGY 146 ( 5 ) S640 - S640 2014.5

A graph-based spoken dialog strategy utilizing multiple understanding hypotheses Reviewed

Norihide Kitaoka, Yuji Kinoshita, Sunao Hara, Chiyomi Miyajima, Kazuya Takeda

Information and Media Technologies 9 ( 1 ) 111 - 120 2014.3

Language：English Publishing type：Research paper (scientific journal) Publisher：Information and Media Technologies Editorial Board

We regarded a dialog strategy for information retrieval as a graph search problem and proposed several novel dialog strategies that can recover from misrecognition through a spoken dialog that traverses the graph. To recover from misrecognition without seeking confirmation, our system kept multiple understanding hypotheses at each turn and searched for a globally optimal hypothesis in the graph whose nodes express understanding states across user utterances in a whole dialog. In the search, we used a new criterion based on efficiency in information retrieval and consistency with understanding hypotheses, which is also used to select an appropriate system response. We showed that our system can make more efficient and natural dialogs than previous ones.

DOI： 10.11185/imt.9.111

New approach to emotional information exchange: Experience metaphor based on life logs Reviewed International journal

Masanobu Abe, Daisuke Fujioka, Kazuto Hamano, Sunao Hara, Rika Mochizuki, Tomoki Watanabe

2014 IEEE International Conference on Pervasive Computing and Communication Workshops, PerCom 2014 Workshops 191 - 194 2014.3

Language：English Publishing type：Research paper (international conference proceedings) Publisher：IEEE

We are striving to develop a new communication technology based on individual experiences that can be extracted from life logs. We have proposed the "Emotion Communication Model" and confirmed that significant correlation exists between experience and emotion. As the second step, particularly addressing impressive places and events, this paper describes an investigation of the extent to which we can share emotional information with others through individuals' experiences. Subjective experiments were conducted using life log data collected during 7-47 months. Experiment results show that (1) impressive places are determined by the distance from home, visit frequency, and direction from home and that (2) positive emotional information is highly consistent among people (71.4%), but it is not true for negative emotional information. Therefore, experiences are useful as metaphors to express positive emotional information.

DOI： 10.1109/PerComW.2014.6815198

Development of a Toolkit Handling Multiple Speech-Oriented Guidance Agents for Mobile Applications Reviewed International journal

Sunao Hara, Hiromichi Kawanami, Hiroshi Saruwatari, Kiyohiro Shikano

Natural Interaction with Robots, Knowbots and Smartphones, Putting Spoken Dialog Systems into Practice 79 - 85 2014

Evaluation of Invalid Input Discrimination Using Bag-of-Words for Speech-Oriented Guidance System Reviewed International journal

Haruka Majima, Rafael Torres, Hiromichi Kawanami, Sunao Hara, Tomoko Matsui, Hiroshi Saruwatari, Kiyohiro Shikano

Natural Interaction with Robots, Knowbots and Smartphones, Putting Spoken Dialog Systems into Practice 389 - 397 2014

A Hybrid Text-to-Speech Based on Sub-Band Approach Reviewed International journal

Takuma Inoue, Sunao Hara, Masanobu Abe

Proceedings of Asia-Pacific Signal and Information Processing Association 2014 Annual Summit and Conference 1 - 4 2014

Language：English Publishing type：Research paper (international conference proceedings) Publisher：IEEE

This paper proposes a sub-band speech synthesis approach to develop high-quality Text-to-Speech (TTS). For the low-frequency band and high-frequency band, Hidden Markov Model (HMM)-based speech synthesis and waveform -based speech synthesis are used, respectively. Both speech synthesis methods are widely known to show good performance and to have benefits and shortcomings from different points of view. One motivation is to apply the right speech synthesis method in the right frequency band. Experiment results show that in terms of the smoothness the proposed approach shows better performance than waveform -based speech synthesis, and in terms of the clarity it shows better than HMM-based speech synthesis. Consequently, the proposed approach combines the inherent benefits from both waveform-based speech synthesis and HMM-based speech synthesis.

DOI： 10.1109/APSIPA.2014.7041575

Invalid Input Rejection Using Bag-of-Words for Speech-oriented Guidance System Reviewed

Haruka Majima, Yoko Fujita, Rafael Torres, Hiromichi Kawanami, Sunao Hara, Tomoko Matsui, Hiroshi Saruwatari, Kiyohiro Shikano

Journal of Information Processing 54 ( 2 ) 443 - 451 2013.2

Language：Japanese

On a real environment speech-oriented information guidance system, a valid and invalid input discrimination is important as invalid inputs such as noise, laugh, cough and utterances between users lead to unpredictable system responses. Generally, acoustic features such as MFCC (Mel-Frequency Cepstral Coefficient) are used for discrimination. Comparing acoustic likelihoods of GMMs (Gaussian Mixture Models) from speech data and noise data is one of the typical methods. In addition to that, using linguistic features, such as speech recognition result, is considered to improve discrimination accuracy as it reflects the task-domain of invalid inputs and meaningless recognition results from noise inputs. In this paper, we introduce Bag-of-Words (BOW) as a feature to discriminate between valid and invalid inputs. Support Vector Machine (SVM) and Maximum Entropy method (ME) are also employed to realize robust classification. We experimented the methods using real environment data obtained from the guidance system "Takemaru-kun." By applying BOW on SVM, the F-measure is improved to 85.09%, from 82.19% when using GMMs. In addition, experiments using features combining BOW with acoustic likelihoods from GMMs, Duration and SNR were conducted, improving the F-measure to 86.58%.

On-line detection of task incompletion for spoken dialog systems based on utterance and behavior tag N-gram Reviewed

Sunao Hara, Norihide Kitaoka, Kazuya Takeda

The IEICE Transactions on Information and Systems (Japanese edition) J96-D ( 1 ) 81 - 93 2013.1

Acoustic Model Training Using Pseudo-Speaker Features Generated by MLLR Transformations for Robust Speaker-Independent Speech Recognition Reviewed

Arata Itoh, Sunao Hara, Norihide Kitaoka, Kazuya Takeda

IEICE TRANSACTIONS on Information and Systems E95D ( 10 ) 2479 - 2485 2012.10

Language：English Publishing type：Research paper (scientific journal) Publisher：IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG

A novel speech feature generation-based acoustic model training method for robust speaker-independent speech recognition is proposed. For decades, speaker adaptation methods have been widely used. All of these adaptation methods need adaptation data. However, our proposed method aims to create speaker-independent acoustic models that cover not only known but also unknown speakers. We achieve this by adopting inverse maximum likelihood linear regression (MLLR) transformation-based feature generation, and then we train our models using these features. First we obtain MLLR transformation matrices from a limited number of existing speakers. Then we extract the bases of the MLLR transformation matrices using PCA. The distribution of the weight parameters to express the transformation matrices for the existing speakers are estimated. Next, we construct pseudo-speaker transformations by sampling the weight parameters from the distribution, and apply the transformation to the normalized features of the existing speaker to generate the features of the pseudo-speakers. Finally, using these features, we train the acoustic models. Evaluation results show that the acoustic models trained using our proposed method are robust for unknown speakers.

DOI： 10.1587/transinf.E95.D.2479

Other Link： http://search.ieice.org/bin/summary.php?id=e95-d_10_2479

Causal analysis of task completion errors in spoken music retrieval interactions Reviewed International journal

Sunao Hara, Norihide Kitaoka, Kazuya Takeda

Proceedings of the 8th international conference on Language Resources and Evaluation (LREC 2012) 1365 - 1372 2012.5

Authorship：Lead author Language：English Publishing type：Research paper (international conference proceedings) Publisher：EUROPEAN LANGUAGE RESOURCES ASSOC-ELRA

In this paper, we analyze the causes of task completion errors in spoken dialog systems, using a decision tree with N-gram features of the dialog to detect task-incomplete dialogs. The dialog for a music retrieval task is described by a sequence of tags related to user and system utterances and behaviors. The dialogs are manually classified into two classes: completed and uncompleted music retrieval tasks. Differences in tag classification performance between the two classes are discussed. We then construct decision trees which can detect if a dialog finished with the task completed or not, using information gain criterion. Decision trees using N-grams of manual tags and automatic tags achieved 74.2% and 80.4% classification accuracy, respectively, while the tree using interaction parameters achieved an accuracy rate of 65.7%. We also discuss more details of the causality of task incompletion for spoken dialog systems using such trees.

Other Link： http://www.lrec-conf.org/proceedings/lrec2012/summaries/1059.html

Robust seed model training for speaker adaptation using pseudo-speaker features generated by inverse CMLLR transformation Reviewed International journal

Arata Itoh, Sunao Hara, Norihide Kitaoka, Kazuya Takeda

Proceedings of 2011 Automatic Speech Recognition and Understanding Workshop (ASRU 2011) 169 - 172 2011.12

Language：English Publishing type：Research paper (international conference proceedings) Publisher：IEEE

In this paper, we propose a novel acoustic model training method which is suitable for speaker adaptation in speech recognition. Our method is based on feature generation from a small amount of speakers' data. For decades, speaker adaptation methods have been widely used. Such adaptation methods need some amount of adaptation data and if the data is not sufficient, speech recognition performance degrade significantly. If the seed models to be adapted to a specific speaker can widely cover more speakers, speaker adaptation can perform robustly. To make such robust seed models, we adopt inverse maximum likelihood linear regression (MLLR) transformation-based feature generation, and then train our seed models using these features. First we obtain MLLR transformation matrices from a limited number of existing speakers. Then we extract the bases of the MLLR transformation matrices using PCA. The distribution of the weight parameters to express the MLLR transformation matrices for the existing speakers is estimated. Next we generate pseudo-speaker MLLR transformations by sampling the weight parameters from the distribution, and apply the inverse of the transformation to the normalized existing speaker features to generate the pseudo-speakers' features. Finally, using these features, we train the acoustic seed models. Using this seed models, we obtained better speaker adaptation results than using simply environmentally adapted models. © 2011 IEEE.

DOI： 10.1109/ASRU.2011.6163925

Training Robust Acoustic Models Using Features of Pseudo-Speakers Generated by Inverse CMLLR Transformations Reviewed International journal

Arata Itoh, Sunao Hara, Norihide Kitaoka, Kazuya Takeda

Proceedings of 2011 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC 2011) 1 - 5 2011.12

Language：English Publishing type：Research paper (international conference proceedings) Publisher：APSIPA

In this paper a novel speech feature generationbased acoustic model training method is proposed. For decades, speaker adaptation methods have been widely used. All existing adaptation methods need adaptation data. However, our proposed method creates speaker-independent acoustic models that cover not only known but also unknown speakers. We do this by adopting inverse maximum likelihood linear regression (MLLR) transformation-based feature generation, and then train our models using these features. First we obtain MLLR transformation matrices from a limited number of existing speakers. Then we extract the bases of the MLLR transformation matrices using PCA. The distribution of the weight parameters to express the MLLR transformation matrices for the existing speakers are estimated. Next we generate pseudo-speaker MLLR transformations by sampling the weight parameters from the distribution, and apply the inverse of the transformation to the normalized existing speaker features to generate the pseudospeakers' features. Finally, using these features, we train the acoustic models. Evaluation results show that the acoustic models which are created are robust for unknown speakers.

On-line detection of task incompletion for spoken dialog systems using utterance and behavior tag N-gram vectors Reviewed International journal

Sunao Hara, Norihide Kitaoka, Kazuya Takeda

Proceedings of the Paralinguistic Information and Its Integration in Spoken Dialogue Systems Workshop 215 - 225 2011.9

Detection of task-incomplete dialogs based on utterance-and-behavior tag N-gram for spoken dialog systems Reviewed International journal

Sunao Hara, Norihide Kitaoka, Kazuya Takeda

Proceedings of INTERSPEECH 2011 1312 - 1315 2011

Authorship：Lead author Language：English Publishing type：Research paper (international conference proceedings) Publisher：ISCA-INT SPEECH COMMUNICATION ASSOC

We propose a method of detecting "task incomplete" dialogs in spoken dialog systems using N-gram-based dialog models. We used a database created during a field test in which inexperienced users used a client-server music retrieval system with a spoken dialog interface on their own PCs. In this study, the dialog for a music retrieval task consisted of a sequence of user and system tags that related their utterances and behaviors. The dialogs were manually classified into two classes: the dialog either completed the music retrieval task or it didn't. We then detected dialogs that did not complete the task, using N-gram probability models or a Support Vector Machine with N-gram feature vectors trained using manually classified dialogs. Off-line and on-line detection experiments were conducted on a large amount of real data, and the results show that our proposed method achieved good classification performance.

Other Link： http://www.isca-speech.org/archive/interspeech_2011/i11_1305.html

Music Recommendation System Based on Human-to-human Conversation Recognition Reviewed International journal

Hiromasa Ohashi, Sunao Hara, Norihide Kitaoka, Kazuya Takeda

Workshop Proceedings of the 7th International Conference on Intelligent Environments: Ambient Intelligence and Smart Environments 10 352 - 361 2011

Language：English Publishing type：Part of collection (book) Publisher：IOS PRESS

We developed an ambient system that plays music suitable for the mood of a human-human conversation using words obtained from a continuous-speech recognition system. Using the correspondence between a document space based on the texts related to the music and an acoustic space that expresses various audio features, the continuous-speech recognition results are mapped to an acoustic space. We performed a subjective evaluation of the system. The subjects rated the recommended music and the result reveals that the 10 most highly recommended selections included suitable music.

DOI： 10.3233/978-1-60750-795-6-352

Automatic detection of task-incompleted dialog for spoken dialog system based on dialog act N-gram Reviewed International journal

Sunao Hara, Norihide Kitaoka, Kazuya Takeda

Proceedings of INTERSPEECH2010 3034 - 3037 2010.9

Authorship：Lead author Language：English Publishing type：Research paper (international conference proceedings) Publisher：ISCA

In this paper, we propose a method of detecting task-incompleted users for a spoken dialog system using an N-gram-based dialog history model. We collected a large amount of spoken dialog data accompanied by usability evaluation scores by users in real environments. The database was made by a field test in which naive users used a client-server music retrieval system with a spoken dialog interface on their own PCs. An N-gram model was trained from sequences that consist of user dialog acts and/or system dialog acts for two dialog classes, that is, the dialog completed the music retrieval task or the dialog incompleted the task. Then the system detects unknown dialogs that is not completed the task based on the N-gram likelihood. Experiments were conducted on large real data, and the results show that our proposed method achieved good classification performance. When the classifier correctly detected all of the task-incompleted dialogs, our proposed method achieved a false detection rate of 6%.

Other Link： http://www.isca-speech.org/archive/interspeech_2010/i10_3034.html

Rapid acoustic model adaptation using inverse MLLR-based feature generation Reviewed International journal

Arata Ito, Sunao Hara, Norihide Kitaoka, Kazuya Takeda

Proceedings of ICA2010 5 1 - 6 2010.8

Language：English Publishing type：Research paper (international conference proceedings)

We propose a technique for generating a large amount of target speaker-like speech features by converting a large amount of prepared speech features of many speakers into features similar to those of the target speaker using a transformation matrix. To generate a large amount of target speaker-like features, the system only needs a very small amount of the target speaker's utterances. This technique enables the system to adapt the acoustic model efficiently from a small amount of the target speaker's utterances. To evaluate the proposed method, we prepared 100 reference speakers and 12 target (test) speakers. We conducted the experiments in an isolated word recognition task using a speech database collected by real PC-based distributed environments and compared our proposed method with MLLR, MAP and the method theoretically equivalent to the SAT. Experimental results proved that the proposed method needed a significantly smaller amount of the target speaker's utterances than conventional MLLR, MAP and SAT.

Estimation method of user satisfaction using N-gram-based dialog history model for spoken dialog system Reviewed International journal

Sunao Hara, Norihide Kitaoka, Kazuya Takeda

LREC 2010 - SEVENTH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION 78 - 83 2010.5

Authorship：Lead author Language：English Publishing type：Research paper (international conference proceedings) Publisher：EUROPEAN LANGUAGE RESOURCES ASSOC-ELRA

In this paper, we propose an estimation method of user satisfaction for a spoken dialog system using an N-gram-based dialog history model. We have collected a large amount of spoken dialog data accompanied by usability evaluation scores by users in real environments. The database is made by a field-test in which naive users used a client-server music retrieval system with a spoken dialog interface on their own PCs. An N-gram model is trained from the sequences that consist of users' dialog acts and/or the system's dialog acts for each one of six user satisfaction levels: from 1 to 5 and phi (task not completed). Then, the satisfaction level is estimated based on the N-gram likelihood. Experiments were conducted on the large real data and the results show that our proposed method achieved good classification performance; the classification accuracy was 94.7% in the experiment on a classification into dialogs with task completion and those without task completion. Even if the classifier detected all of the task incomplete dialog correctly, our proposed method achieved the false detection rate of only 6%.

Other Link： http://www.lrec-conf.org/proceedings/lrec2010/summaries/579.html

Data collection and usability study of a PC-based speech application in various user environments Reviewed International journal

Sunao Hara, Chiyomi Miyajima, Katsunobu Ito, Norihide Kitaoka, Kazuya Takeda

Proceedings of Oriental-COCOSDA 2008 39 - 44 2008.11

In-car Speech Data Collection along with Various Multimodal Signals Reviewed International journal

Akira Ozaki, Sunao Hara, Takashi Kusakawa, Chiyomi Miyajima, Takanori Nishino, Norihide Kitaoka, Katunobu Itou, Kazuya Takeda

Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08) 1846 - 1851 2008.5

Language：English Publishing type：Research paper (international conference proceedings) Publisher：EUROPEAN LANGUAGE RESOURCES ASSOC-ELRA

In this paper, a large-scale real-world speech database is introduced along with other multimedia driving data. We designed a data collection vehicle equipped with various sensors to synchronously record twelve-channel speech, three-channel video, driving behavior including gas and brake pedal pressures, steering angles, and vehicle velocities, physiological signals including driver heart rate, skin conductance, and emotion-based sweating on the palms and soles, etc. These multimodal data are collected while driving on city streets and expressways under four different driving task conditions including two kinds of monologues, human-human dialog, and human-machine dialog. We investigated the response timing of drivers against navigator utterances and found that most overlapped with the preceding utterance due to the task characteristics and the features of Japanese. When comparing utterance length, speaking rate, and the filler rate of driver utterances in human-human and human-machine dialogs, we found that drivers tended to use longer and faster utterances with more fillers to talk with humans than machines.

Other Link： http://www.lrec-conf.org/proceedings/lrec2008/summaries/472.html

Data Collection System for the Speech Utterances to an Automatic Speech Recognition System under Real Environments Reviewed

Sunao Hara, Chiyomi Miyajima, Katsunobu Itou, Kazuya Takeda

The IEICE transactions on information and systems J90-D ( 10 ) 2807 - 2816 2007.10

An online customizable music retrieval system with a spoken dialogue interface Reviewed International journal

Sunao Hara, Chiyomi Miyajima, Katsunobu Itou, Kazuya Takeda

The Journal of the Acoustical Society of America 120 ( 5-2 ) 3378 - 3379 2006.11

Novel orally active alpha 4 integrin antagonist, T-728, attenuates dextran sodium sulfate-induced chronic colitis in mice

Ken-Ichi Hayashi, Hiroyuki Meguro, Sunao Hara, Rie Sasaki, Yoko Koga, Meiko Takeshita, Naoyoshi Yamamoto, Hiroe Hirokawa, Mie Kainoh

GASTROENTEROLOGY 130 ( 4 ) A352 - A352 2006.4

Preliminary Study of a Learning Effect on Users to Develop a New Evaluation of the Spoken Dialogue System Reviewed International journal

Sunao Hara, Ayako Shirose, Chiyomi Miyajima, Katsunobu Ito, Kazuya Takeda

Proceedings of Oriental-COCOSDA 2005 164 - 168 2005.12

　 PREV - NEXT 　

1