上下文語言模型化技術於閱讀理解之研究

A Study on Contextualized Language Modeling for Machine Reading Comprehension

隨著深度學習的發展，機器閱讀理解的研究已有了長足的進步，並在許多實際應用情境上嶄露頭角。機器閱讀理解是一項用於評估機器對語言的理解能力的自然語言處理任務，其形式為：給定文章段落與相關問題，電腦自動根據文章段落與相關問題來進行回答。本研究嘗試使用兩種以BERT為基礎的預訓練語言模型：BERT-wwm和MacMERT，發展能夠達到更加預測表現的機器閱讀理解方法。此外，考慮到閱讀理解中的文章類型可能對於回答模式有潛在影響，我們針對訓練資料集的文章段落進行分群，以此作為額外資訊結合到語言模型的輸入。另一方面，我們也探索使用集成學習法來結合上述兩種預訓練語言模型，以進一步提升機器閱讀理解的表現。

With the recent breakthrough of deep learning technologies, research on machine reading comprehension (MRC) has attracted much attention and found its versatile applications in many use cases. MRC is an important natural language processing (NLP) task aiming to assess the ability of a machine to understand natural language expressions, which is typically operationalized by first asking questions based on a given text paragraph and then receiving machine-generated answers in accordance with the given context paragraph and questions. In this paper, we leverage two novel pretrained language models built on top of Bidirectional Encoder Representations from Transformers (BERT), namely BERT-wwm and MacBERT, to develop effective MRC methods. In addition, we also seek to investigate whether additional incorporation of the categorical information about a context paragraph can benefit MRC or not, which is achieved based on performing context paragraph clustering on the training dataset. On the other hand, an ensemble learning approach is proposed to harness the synergistic power of the aforementioned two BERT-based models so as to further promote MRC performance.