正體中文斷詞系統應用於大型語料庫之多方評估研究

Multifaceted Assessments of Traditional Chinese Word Segmentation Tool on Large Corpora

本研究之目的在於運用多種數值指標及實驗資料來評估CKIP、Jieba、MONPA等三種廣泛應用於臺灣自然語言處理產學界的正體中文斷詞器。我們特別針對運算效能、資源需求等等面向，檢驗其應用於大型語言文字資料集時，處理斷詞、詞性標註及命名實體辨識等工作之成效。實驗結果顯示，MONPA利用圖形運算加速器（GPU）進行批次處理斷詞時，可以大幅度縮減巨量中文資料的運算時間，且其斷詞、詞性標註、命名實體辨識等多功能標籤均達到令人滿意的品質，且其產出之標註結果可有效輔助提升中文自然語言處理的後續相關任務成效。

This study aims to evaluate three most popular word segmentation tool for a large Traditional Chinese corpus in terms of their efficiency, resource consumption, and cost. Specifically, we compare the performances of Jieba, CKIP, and MONPA on word segmentation, part-of-speech tagging and named entity recognition through extensive experiments. Experimental results show that MONPA using GPU for batch segmentation can greatly reduce the processing time of massive datasets. In addition, its features such as word segmentation, part-of-speech tagging, and named entity recognition are beneficial to downstream applications.