Faculty Profiles - HARA Sunao

Papers - HARA Sunao

Division display 1 - 20 of about 62 ／ All the affair displays >>

Continual learning on audio scene classification using representative data and memory replay GANs Reviewed International coauthorship International journal

Ibnu Daqiqil ID, Masanobu Abe, Sunao Hara

Bulletin of Electrical Engineering and Informatics 14 ( 1 ) 568 - 580 2025.2

Authorship：Last author Language：English Publishing type：Research paper (scientific journal) Publisher：Institute of Advanced Engineering and Science

This paper proposes a methodology aimed at resolving catastropic forgetting problem by choosing a limited portion of the historical dataset to act as a representative memory. This method harness the capabilities of generative adversarial networks (GANs) to create samples that expand upon the representative memory. The main advantage of this method is that it not only prevents catastrophic forgetting but also improves backward transfer and has a relatively stable and small size. The experimental results show that combining real representative data with artificially generated data from GANs, yielded better outcomes and helped counteract the negative effects of catastrophic forgetting more effectively than solely relying on GAN-generated data. This mixed approach creates a richer training environment, aiding in the retention of previous knowledge. Additionally, when comparing different methods for selecting data as the proportion of GAN-generated data increases, the low probability and mean cluster methods performed the best. These methods exhibit resilience and consistency by selecting more informative samples, thus improving overall performance.

DOI： 10.11591/eei.v14i1.8127

researchmap

OtologMap: a case study on the construction of an environmental sound map recorded by smart devices at Okayama and Kurashiki Reviewed International journal

Sunao Hara, Masanobu Abe

Noise Control Engineering Journal 1 - 12 2025.2

Explicit Prosody Control to Realize Discourse Focus in End-to-End Text-to-Speech Reviewed International journal

Takumi WADA, Sunao HARA, Masanobu ABE

IEEE International Workshop on Machine Learning for Signal Processing 2024.9

Speech-Emotion Control for Text-to-Speech in Spoken Dialogue Systems Using Voice Conversion and x-vector Embedding Reviewed International journal

Shunichi Kohara, Masanobu Abe, Sunao Hara

2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC 2023) 2280 - 2286 2023.11

Language：English Publishing type：Research paper (international conference proceedings)

In this paper, we propose an algorithm to control both speaker individuality and emotional expressions in synthesized speech, where the most important feature is the controllability of intensity in emotional expressions. An aim of the proposed algorithm is to generate various responses including emotions in text-to-speech (TTS) for spoken dialogue systems (SDS), which results in making the system more human-like. An idea is to control emotion and its intensity in line with the user's utterances. For example, when a user happily talks to SDS, the agent of the SDS responses with happy voice. Generally, voice quality of a user and the agent are different. Therefore, the proposed algorithm consists of two steps: (1) voice conversion to change speaker individuality including emotional expressions and (2) TTS with x-vector acting as an embedding vector to mainly control speech quality related to the intensity of emotions. Evaluation experiments are carried out using a scenario of a spoken dialogue system, where a teacher system of TTS encourages or cheers up students according to students' utterances. The experiment results showed that TTS can successfully reproduce the emotion and its intensity that are extracted from students' utterances, while maintaining the teacher's speaker individuality.

DOI： 10.1109/APSIPAASC58517.2023.10317413

Scopus

researchmap

Speech Synthesis Using Ambiguous Inputs From Wearable Keyboards Reviewed International journal

Matsuri Iwasaki, Sunao Hara, Masanobu Abe

2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC 2023) 1172 - 1178 2023.11

Language：English Publishing type：Research paper (international conference proceedings)

This paper proposes a new application in speech communication using text-to-speech (TTS), and the goal is to enable dysarthria, articulation disorder, or persons who have difficulty in speaking to communicate anywhere and anytime using speech to express their thoughts and feelings. To achieve this goal, an input method is required. Thus, we propose a new text-entry method based on three concepts. First, from an easy-to-carry perspective, we used a wearable keyboard that inputs digits from 0 to 9 in decimal notation according to 10-finger movements. Second, from a no-training perspective, users input sentences in a way of touch typing using the wearable keyboard. Following this method, we obtained a sequence of numbers corresponding to the sentence. Third, a neural machine translation (NMT) method is applied to estimate texts from the sequence of numbers. The NMT was trained using two datasets; one is a Japanese-English parallel corpus containing 2.8 million pairs of sentences, which were extracted from TV and movie subtitles, while the other is a Japanese text dataset containing 32 million sentences, which were extracted from a question-and-answer platform. Using the model, phonemes and accent symbols were estimated from a sequence of numbers. Thus, the result accuracy in symbol levels was 91.48% and 43.45% of all the sentences were completely estimated with no errors. To subjectively evaluate feasibility of the NMT model, a two-person word association game was conducted; one gave hints using synthesized speech that is generated from symbols estimated by NMT, while the other guessed answers. As a result, 67.95% of all the quizzes were correctly answered, and experiment results show that the proposed method has the potential for dysarthria to communicate with TTS using a wearable keyboard.

DOI： 10.1109/APSIPAASC58517.2023.10317228

Scopus

researchmap

Sound map of urban areas recorded by smart devices: case study at Okayama and Kurashiki Invited International journal

Sunao Hara, Masanobu Abe

Proceedings of the 52nd International Congress and Exposition on Noise Control Engineering (Inter-Noise 2023) 1 - 12 2023.8

Predictions for sound events and soundscape impressions from environmental sound using deep neural networks Invited International journal

Sunao Hara, Masanobu Abe

Proceedings of the 52nd International Congress and Exposition on Noise Control Engineering (Inter-Noise 2023) 1 - 12 2023.8

Prediction method of Soundscape Impressions using Environmental Sounds and Aerial Photographs Reviewed International journal

Yusuke Ono, Sunao Hara, Masanobu Abe

2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) 1222 - 1227 2022.11

Incremental Audio Scene Classifier Using Rehearsal-Based Strategy Reviewed International journal

Ibnu Daqiqil Id, Masanobu Abe, Sunao Hara

2022 IEEE 10th Global Conference on Consumer Electronics (GCCE) 69 - 623 2022.10

Speech-Like Emotional Sound Generation Using WaveNet Reviewed

Kento Matsumoto, Sunao Hara, Masanobu Abe

IEICE Transactions on Information and Systems E105.D ( 9 ) 1581 - 1589 2022.9

Language：English Publishing type：Research paper (scientific journal) Publisher：Institute of Electronics, Information and Communications Engineers (IEICE)

In this paper, we propose a new algorithm to generate Speech-like Emotional Sound (SES). Emotional expressions may be the most important factor in human communication, and speech is one of the most useful means of expressing emotions. Although speech generally conveys both emotional and linguistic information, we have undertaken the challenge of generating sounds that convey emotional information alone. We call the generated sounds "speech-like," because the sounds do not contain any linguistic information. SES can provide another way to generate emotional response in human-computer interaction systems. To generate "speech-like" sound, we propose employing WaveNet as a sound generator conditioned only by emotional IDs. This concept is quite different from the WaveNet Vocoder, which synthesizes speech using spectrum information as an auxiliary feature. The biggest advantage of our approach is that it reduces the amount of emotional speech data necessary for training by focusing on non-linguistic information. The proposed algorithm consists of two steps. In the first step, to generate a variety of spectrum patterns that resemble human speech as closely as possible, WaveNet is trained with auxiliary mel-spectrum parameters and Emotion ID using a large amount of neutral speech. In the second step, to generate emotional expressions, WaveNet is retrained with auxiliary Emotion ID only using a small amount of emotional speech. Experimental results reveal the following: (1) the two-step training is necessary to generate the SES with high quality, and (2) it is important that the training use a large neutral speech database and spectrum information in the first step to improve the emotional expression and naturalness of SES.

DOI： 10.1587/transinf.2021edp7236

Web of Science

researchmap

Other Link： https://www.webofscience.com/wos/woscc/full-record/WOS:000852731400008

Concept drift adaptation for audio scene classification using high-level features Reviewed International journal

Ibnu Daqiqil Id, Masanobu Abe, Sunao Hara

2022 IEEE International Conference on Consumer Electronics (ICCE) 2022.1

Acoustic Scene Classifier Based on Gaussian Mixture Model in the Concept Drift Situation Reviewed International journal

Ibnu Daqiqil Id, Masanobu Abe, Sunao Hara

Advances in Science, Technology and Engineering Systems Journal 6 ( 5 ) 167 - 176 2021.9

Phonetic and Prosodic Information Estimation from Texts for Genuine Japanese End-to-End Text-to-Speech Reviewed International journal

Naoto Kakegawa, Sunao Hara, Masanobu Abe, Yusuke Ijima

Interspeech 2021 126 - 130 2021.8

Language：English Publishing type：Research paper (international conference proceedings) Publisher：ISCA

The biggest obstacle to develop end-to-end Japanese text-to-speech (TTS) systems is to estimate phonetic and prosodic information (PPI) from Japanese texts. The following are the reasons: (1) the Kanji characters of the Japanese writing system have multiple corresponding pronunciations, (2) there is no separation mark between words, and (3) an accent nucleus must be assigned at appropriate positions. In this paper, we propose to solve the problems by neural machine translation (NMT) on the basis of encoder-decoder models, and compare NMT models of recurrent neural networks and the Transformer architecture. The proposed model handles texts on token (character) basis, although conventional systems handle them on word basis. To ensure the potential of the proposed approach, NMT models are trained using pairs of sentences and their PPIs that are generated by a conventional Japanese TTS system from 5 million sentences. Evaluation experiments were performed using PPIs that are manually annotated for 5,142 sentences. The experimental results showed that the Transformer architecture has the best performance, with 98.0% accuracy for phonetic information estimation and 95.0% accuracy for PPI estimation. Judging from the results, NMT models are promising toward end-to-end Japanese TTS.

DOI： 10.21437/interspeech.2021-914

Web of Science

researchmap

Model architectures to extrapolate emotional expressions in DNN-based text-to-speech Reviewed International journal

Katsuki Inoue, Sunao Hara, Masanobu Abe, Nobukatsu Hojo, Yusuke Ijima

Speech Communication 126 35 - 43 2021.2

Language：English Publishing type：Research paper (scientific journal) Publisher：Elsevier BV

This paper proposes architectures that facilitate the extrapolation of emotional expressions in deep neural network (DNN)-based text-to-speech (TTS). In this study, the meaning of “extrapolate emotional expressions” is to borrow emotional expressions from others, and the collection of emotional speech uttered by target speakers is unnecessary. Although a DNN has potential power to construct DNN-based TTS with emotional expressions and some DNN-based TTS systems have demonstrated satisfactory performances in the expression of the diversity of human speech, it is necessary and troublesome to collect emotional speech uttered by target speakers. To solve this issue, we propose architectures to separately train the speaker feature and the emotional feature and to synthesize speech with any combined quality of speakers and emotions. The architectures are parallel model (PM), serial model (SM), auxiliary input model (AIM), and hybrid models (PM&AIM and SM&AIM). These models are trained through emotional speech uttered by few speakers and neutral speech uttered by many speakers. Objective evaluations demonstrate that the performances in the open-emotion test provide insufficient information. They make a comparison with those in the closed-emotion test, but each speaker has their own manner of expressing emotion. However, subjective evaluation results indicate that the proposed models could convey emotional information to some extent. Notably, the PM can correctly convey sad and joyful emotions at a rate of 60%.

DOI： 10.1016/j.specom.2020.11.004

Web of Science

researchmap

Other Link： https://arxiv.org/abs/2102.10345

Module Comparison of Transformer-TTS for Speaker Adaptation based on Fine-tuning Reviewed International journal

Katsuki Inoue, Sunao Hara, Masanobu Abe

Proceedings of APSIPA Annual Summit and Conference (APSIPA-ASC 2020) 826 - 830 2020.12

Language：English Publishing type：Research paper (international conference proceedings) Publisher：IEEE/APSIPA

End-to-end text-to-speech (TTS) models have achieved remarkable results in recent times. However, the model requires a large amount of text and audio data for training. A speaker adaptation method based on fine-tuning has been proposed for constructing a TTS model using small scale data. Although these methods can replicate the target speaker s voice quality, synthesized speech includes the deletion and/or repetition of speech. The goal of speaker adaptation is to change the voice quality to match the target speaker ' s on the premise that adjusting the necessary modules will reduce the amount of data to be fine-tuned. In this paper, we clarify the role of each module in the Transformer-TTS process by not updating it. Specifically, we froze character embedding, encoder, layer predicting stop token, and loss function for estimating sentence ending. The experimental results showed the following: (1) fine-tuning the character embedding did not result in an improvement in the deletion and/or repetition of speech, (2) speech deletion increases if the encoder is not fine-tuned, (3) speech deletion was suppressed when the layer predicting stop token is not fine-tuned, and (4) there are frequent speech repetitions at sentence end when the loss function estimating sentence ending is omitted.

Web of Science

researchmap

Concept Drift Adaptation for Acoustic Scene Classifier Based on Gaussian Mixture Model Reviewed

Ibnu Daqiqil Id, Masanobu Abe, Sunao Hara

Proceedings of IEEE REGION 10 CONFERENCE (TENCON 2020) 450 - 455 2020.11

Controlling the Strength of Emotions in Speech-Like Emotional Sound Generated by WaveNet Reviewed International journal

Kento Matsumoto, Sunao Hara, Masanobu Abe

Proceedings of Interspeech 2020 3421 - 3425 2020.10

Semi-Supervised Speaker Adaptation for End-to-End Speech Synthesis with Pretrained Models Reviewed International coauthorship International journal

Katsuki Inoue, Sunao Hara, Masanobu Abe, Tomoki Hayashi, Ryuichi Yamamoto, Shinji Watanabe

Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2020) 7634 - 7638 2020.5

Language：English Publishing type：Research paper (international conference proceedings) Publisher：IEEE

Recently, end-to-end text-to-speech (TTS) models have achieved a remarkable performance, however, requiring a large amount of paired text and speech data for training. On the other hand, we can easily collect unpaired dozen minutes of speech recordings for a target speaker without corresponding text data. To make use of such accessible data, the proposed method leverages the recent great success of state-of-the-art end-to-end automatic speech recognition (ASR) systems and obtains corresponding transcriptions from pretrained ASR models. Although these models could only provide text output instead of intermediate linguistic features like phonemes, end-to-end TTS can be well trained with such raw text data directly. Thus, the proposed method can greatly simplify a speaker adaptation pipeline by consistently employing end-to-end ASR/TTS ecosystems. The experimental results show that our proposed method achieved comparable performance to a paired data adaptation method in terms of subjective speaker similarity and objective cepstral distance measures.

DOI： 10.1109/icassp40776.2020.9053371

Web of Science

researchmap

DNN-based Voice Conversion with Auxiliary Phonemic Information to Improve Intelligibility of Glossectomy Patients' Speech Reviewed International journal

Hiroki Murakami, Sunao Hara, Masanobu Abe

Proceedings of APSIPA Annual Summit and Conference 2019 138 - 142 2019.11

Authorship：Corresponding author Language：English Publishing type：Research paper (international conference proceedings) Publisher：IEEE

In this paper, we propose using phonemic information in addition to acoustic features to improve the intelligibility of speech uttered by patients with articulation disorders caused by a wide glossectomy. Our previous studies showed that voice conversion algorithm improves the quality of glossectomy patients' speech. However, losses in acoustic features of glossectomy patients' speech are so large that the quality of the reconstructed speech is low. To solve this problem, we explored potentials of several additional information to improve speech intelligibility. One of the candidates is phonemic information, more specifically Phoneme Labels as Auxiliary input (PLA). To combine both acoustic features and PLA, we employed a DNN-based algorithm. PLA is represented by a kind of one-of-k vector, i.e., PLA has a weight value (<1.0) that gradually changes in time axis, whereas one-of-k has a binary value (0 or 1). The results showed that the proposed algorithm reduced the mel-frequency cepstral distortion for all phonemes, and almost always improved intelligibility. Notably, the intelligibility was largely improved in phonemes /s/ and /z/, mainly because the tongue is used to sustain constriction to produces these phonemes. This indicates that PLA works well to compensate the lack of a tongue.

DOI： 10.1109/APSIPAASC47483.2019.9023168

Web of Science

researchmap

Speech-like Emotional Sound Generator by WaveNet Reviewed International journal

Kento Matsumoto, Sunao Hara, Masanobu Abe

Proceedings of APSIPA Annual Summit and Conference 2019 143 - 147 2019.11

Language：English Publishing type：Research paper (international conference proceedings) Publisher：IEEE

In this paper, we propose a new algorithm to generate Speech-like Emotional Sound (SES). Emotional information plays an important role in human communication, and speech is one of the most useful media to express emotions. Although, in general, speech conveys emotional information as well as linguistic information, we have undertaken the challenge to generate sounds that convey emotional information without linguistic information, which results in making conversations in human-machine interactions more natural in some situations by providing non-verbal emotional vocalizations. We call the generated sounds "speech-like", because the sounds do not contain any linguistic information. For the purpose, we propose to employ WaveNet as a sound generator conditioned by only emotional IDs. The idea is quite different from WaveNet Vocoder that synthesizes speech using spectrum information as auxiliary features. The biggest advantage of the idea is to reduce the amount of emotional speech data for the training. The proposed algorithm consists of two steps. In the first step, WaveNet is trained to obtain phonetic features using a large speech database, and in the second step, WaveNet is re-trained using a small amount of emotional speech. Subjective listening evaluations showed that the SES could convey emotional information and was judged to sound like a human voice.

DOI： 10.1109/APSIPAASC47483.2019.9023346

Web of Science

researchmap

　 PREV - NEXT 　

< HARA Sunao >

Papers - HARA Sunao