通過半監督學習改進端到端台語語音至中文文字翻譯)

Improving End-to-end Taiwanese-Speech-to-Chinese-Text Translation by Semi-supervised Learning

傳統台語語音辨識的主要問題，為缺乏大量且公開的台語語料集，以及台語文字書寫系統不統一；前者導致進行語音辨識的任務上面臨資料不足，而後者則造成輸出格式不統一且不易讀解。因此，本研究以台語語音至中文文字的語音翻譯為任務，透過預訓練語音模型結合端到端深度學習模型的架構，來建立台語語音至中文文字的語音翻譯模型。我們的方法是以少量台語語音配對中文文本的語料為基礎，並透過大量蒐集未配對的台語語音資料，並設計各種演算法來利用大量未配對語料改善台語語音至中文文字的翻譯系統。研究探討主要分為端到端語音翻譯模型、預訓練語音模型特徵、疊代訓練方法以及語料清洗四種改進方向。根據實驗結果顯示，上述方法皆能有效改善台語語音至中文文字的翻譯表現。

The main challenges in Taiwanese speech recognition are the lack of abundant and publicly available Taiwanese speech corpora, and the inconsistency in the written system of Taiwanese. The former results in insufficient data for speech recognition tasks, while the latter leads to inconsistent output formats and difficulties in interpretation. Therefore, this study takes the speech translation from Taiwanese speech to Chinese text as the task, and builds a speech translation model from Taiwanese speech to Chinese text by combining the pre-trained speech model with the architecture of the end-to-end deep learning model. Our method is based on a small amount of Taiwanese speech paired with Chinese text, and by collecting a large amount of unpaired Taiwanese speech data, and designing various algorithms to use a large amount of unpaired corpus to improve the system of translating Taiwanese speech into Chinese text. The research and discussion are mainly divided into four improvement directions: end-to-end speech translation model, pre-trained speech model features, iterative training method and corpus cleaning. Experimental results show that the above methods can effectively improve the translation performance of Taiwanese speech to Chinese text.