I am currently pursuing my Ph.D. in informatics at Kawahara Lab,
Department of Intelligence Science and Technology, Graduate School of Informatics, Kyoto University.
My research interests lie in the field of computer assisted language learning (CALL). The primary focus of my research is to make use of state-of-the-art technologies in order to assist language learners and to facilitate the process of teaching and learning a foreign language. At the present time my research focuses on developing a novel technique of captioning for second language learners for which I have received a scholarship from the Ministry of Education, Culture, Sport, Science and Technology of Japan (MEXT).
Recently working on "Partial and Synchronized Caption" project in order to advance the sytem and incorporate it into a CALL system. This on-going study investigates use of a novel method of captioning, PSC, as a listening tool for second language (L2) learners. In this method, the term partial and synchronized caption pertains to the presence of only a selected set of words in a caption where words are synced to their corresponding speech signals. This new approach relies on the latest advances in speech recognition technology; where an automatic speech recognition system (Julius 4.3) was trained using the desired corpora to precisely align each word to its corresponding speech signal. This word-level alignment emulates the speech flow and allows for text-to-speech mapping. The outcome of this process is used to generate partial captions by automatically selecting words and phrases which are likely to hinder learner's listening comprehension. The selected words are presented in caption while the rest are masked by dots in order to make comprehension based more on listening to the speech rather than solely on reading the caption text. The criteria for selection are defined by three features: "speech rate", "word frequency" and "specificity". This method is based on the premise that occurrence of infrequent words in listening materials and fast delivery of speech by the speaker attenuate L2 listening comprehension. In this view PSC strives to assist the learners by presenting these problematic words/phrases in the caption. For effective word selection, learner's vocabulary size and tolerable rate of speech were adopted as the basis for generating the captions. Finally PSCs are automatically prepared for learners at three proficiency levels: "beginners", "pre-intermediates" and "intermediates". This type of captioning is anticipated to be not only an assistive tool to enhance L2 learners' listening comprehension skills, but also a medium to decrease dependence on captions thus preparing learners for real-world situations. Click here (the official project page) to see demonstration videos on how PSC works.
* These works are cited in Intralingual Subtitles (Captions) and Foreign/Second Language Acquisition Research Bibliography by Professor Gunter Burgers.