臺灣客語斷詞前導研究與模型建立

The Pilot Study and Model Construction for Word Segmentation in Taiwan Hakka

斷詞是自然語言處理以及資料檢索查詢的關鍵角色，本文《臺灣客語語料庫》的斷詞前導研究，乃是運用華語斷詞系統Stanford CoreNLP，以華客對應轉換方式為客語文本進行斷詞與標記。然而，斷詞效能不盡理想，因華客之間有許多字詞難以對譯，且臺灣客語次方言之間詞彙與語音也存在差異。有鑑於此，臺灣客語語料庫提出客語專屬的斷詞模型，建構客語詞庫，以六腔分列詞目，並採用長詞優先演算法以及動態規劃演算法設計。斷詞效能評估測試結果顯示，詞庫查找與詞頻統計（透過長詞優先演算法以及N-gram語言模型）兩者相輔相成，無論是斷詞效能或是斷詞準確率皆有著明顯提升。

Word segmentation plays a key role in natural language processing and data retrieval queries. The pilot study employed Stanford CoreNLP, a word segmentation system for Chinese, for segmentation and tagging of Hakka texts in Taiwan Hakka Corpus. Nevertheless, the performance was unsatisfactory due to the intractable correspondent translations between Mandarin and Hakka and the lexical and phonetic varieties among the six dialects of Hakka. In view of these reasons, a tailormade Hakka segmentation model is constructed that encompasses Hakka lexicon with six accents and that applies Maximum Matching Algorithm (MM) and dynamic programming algorithm in the system. The segmentation performance evaluation test results show that combining lexicon lookup and word frequency statistics (with Maximum Matching Algorithm and N-gram Language Model) significantly improves both segmentation performance and accuracy.