運用基於生成預訓練轉換器架構的OpenAI Whisper多語言語音辨識引擎之台語及華語語音辨識之實作

Taiwanese/Mandarin Speech Recognition using OpenAI's Whisper Multilingual Speech Recognition Engine Based on Generative Pretrained Transformer Architecture

本篇論文中，我們對OpenAI Whisper進行台語的模型微調，使Whisper能夠輸出華語和台語的繁體漢字。我們使用Hugging Face官方所提供的Whisper的Medium和Large-v2模型和微調方式，並使用CommonVoice的台語資料集和網路上蒐集的台語連續劇影片和字幕檔共800小時，CER最佳為50.7%。我們將在後續提供我們所微調的程式碼。

In this paper, we conducted model finetuning on OpenAI's Whisper for Taiwanese languages, enabling Whisper to generate both Mandarin and Taiwanese text outputs. We employed Hugging Face's official Whisper models, namely Medium and Large-v2, and their finetuning methodology. Additionally, we utilized the Taiwanese dataset from CommonVoice and collected around 800 hours of Taiwanese drama videos along with their subtitle files from the internet. The achieved Character Error Rate (CER) reached approximately 50.7%. We will provide the code we have fine-tuned in the subsequent updates.