運用Python結合語音辨識及合成技術於自動化音文同步之實作

A Python Implementation of Automatic Speech-text Synchronization Using Speech Recognition and Text-to-Speech Technology

本研究設計一個方便處理有聲書音文同步的技術，利用雲端的文字轉語音（Text-to-speech）技術，結合語音辨識（Speech Recognition）技術，讓使用者能夠使用自行準備的文章來製作自己的『跟述練習』（Shadowing technique）的學習素材，製作達到詞層級（Word-level）的音文同步有聲書。此音文同步有聲書是藉由『帶時間點的文字』（Timed-text）檔案所製作，而帶時間點的文字則是由使用者所提供的文章連同對應的語音聲波檔案，經由一套名為CGUAlign的音文同步技術之處理所產生的。CGUAlign是運用Python將一有名的語音辨識技術─HTK（Hidden Markov Model Toolkit）包裝，只要提供文字檔及其朗讀的語音檔，其中語音檔是經由雲端語音合成技術而得來的，即能製作出音文同步的帶時間點的文字檔案，隨後，我們也建立一個簡易的以JavaScript製作的網站，能夠運用這個檔案做電腦輔助語言學習（Computer-assisted language learning, CALL）之用，此網站能夠閱讀音文同步有聲書，讓使用者能夠較輕鬆的做跟述練習，最後我們也提供即時翻譯的功能來達到電腦輔助語言學習的目標。

In this study, we establish a method to create speech and text synchronized audiobooks with “speech recognition” and “cloud text-to-speech” technology. The user can prepare his own arbitrary articles to create the learning materials for 'Shadowing technique' with this method. Besides, the materials are made by 'word-level' speech and text synchronized audiobooks. These audiobooks are created by 'timed-text' files, and the files are produced from the user's articles and corresponding speech files. By synchronization for speech and text technology, named 'CGUAlign', user can easily make the 'Timed-text' files. CGUAlign, uses Python to wrap the well-known speech recognition technology─HTK(Hidden Markov Model Toolkit). Just providing text file and the corresponding speech file, obtained from cloud text-to-speech technology, CGUAlign can create the timed-text file to achieve the synchronization of speech and text. Subsequently, we also build a simple website created with JavaScript. This website can use the timed-text file as CALL(Computer-assisted Language Learning) purposes. Using the website, user can browse the synchronized audiobooks to easily do Shadowing technique. Finally this website also provides dictionary function to achieve the goal of CALL.