Mitigating Impacts of Word Segmentation Errors on Collocation Extraction in Chinese

Mitigating Impacts of Word Segmentation Errors on Collocation Extraction in Chinese

隨著網路的盛行，自動斷詞與標記的大規模語料庫逐漸普及。自動化不可避免地引入一些斷詞與標記的錯誤，並可能對下游任務產生負面影響。搭配詞的自動抽取是一項受斷詞品質影響的任務。本文探討一些方法試圖減輕斷詞錯誤對漢語搭配詞抽取之影響。我們嘗試了一個結合多個共現訊息的簡單線性模型，試圖減少抽取出之搭配詞含有的斷詞錯誤。實驗結果顯示，此模型無法區分搭配詞是否為斷詞錯誤所導致。因此，我們使用了FastText詞向量的訊息進行了另一個案例研究。結果顯示，由斷詞錯誤所產生的假搭配詞與真正的搭配詞，其之間的語義相似性具有不同的特徵。未來研究可嘗試在搭配詞抽取中加入詞向量的訊息。

The prevalence of the web has brought about the construction of many large-scale, automatically segmented and tagged corpora, which inevitably introduces errors due to automation and are likely to have negative impacts on downstream tasks. Collocation extraction from Chinese corpora is one such task that is profoundly influenced by the quality of word segmentation. This paper explores methods to mitigate the negative impacts of word segmentation errors on collocation extraction in Chinese. In particular, we experimented with a simple model that aims to combine several association measures linearly to avoid retrieving false collocations resulting from word segmentation errors. The results of the experiment show that this simple model could not differentiate between true collocations and false collocations resulting from word segmentation errors. An ad hoc case study incorporating information from FastText word vectors is also conducted. The results show that collocates resulting from correct and erroneous word segmentation have different profiles in terms of the semantic similarities between the collocates. The incorporation of word vector information to differentiate between true and false collocations is suggested for future work.