利用統計方法及中文訓練資料處理台語文詞性標記

Modeling Taiwanese POS tagging with statistical methods and Mandarin training data

本文提出利用有六萬多詞條的台華辭典以及千萬詞的中文訓練資料來做台語文詞性標記的方法。台語文語料為包括全羅馬字及漢羅合用兩種書寫文本的文學資料，文類涵蓋散文、小說、劇本等，詞類集採用中央研究院詞庫小組所訂定的中文詞類集。我們開發語詞對齊檢查程式，將兩種文本的語料逐詞對齊，透過台華辭典查詢每個語詞相對應的中文候選詞，接著利用中文訓練資料，以HMM機率模型挑選出最適當的中文對譯詞，再以MEMM分類器標記詞性。實驗結果顯示，以此方法做台語文詞性標記，我們得到91.49%的正確率，並針對標記錯誤分析其原因。以此基礎，我們也得到了初步的台語文訓練語料。

In this paper, we propose a POS tagging method using more than 60 thousand entries of Taiwanese-Mandarin translation dictionary and 10 million words of Mandarin training data to tag Taiwanese. The literary written Taiwanese corpora have both Romanization script and Han-Romanization mixed script, the genre includes prose, fiction and drama. We follow tagset drawn up by CKIP. We develop word alignment checker to help the two scripts word alignment work, and then lookup Taiwanese-Mandarin translation dictionary to find the corresponding Mandarin candidate words, select the most suitable Mandarin word using HMM probabilistic model from the Mandarin training data, and finally tag the word using MEMM classifier. We achieve an accuracy rate of 91.49% on Taiwanese POS tagging work, and analysis the errors. We also get the preliminary Taiwanese training data.