應用多模式特徵融合的深度注意力網路進行謠言檢測

Rumor Detection Using Deep Attention Networks With Multimodal Feature Fusion

隨著社群平台蓬勃的發展，許多謠言與假訊息也充斥在社群媒體之中。現今各大社群平台大多是透過人工的舉報或統計的方式來進行謠言的分辨，這在資訊快速傳播的時代，非常缺乏效率。本論文提出一個結合圖像描述模型的多模式特徵融合方法，並透過深度注意力網路來進行謠言檢測。從Tweets中擷取出圖像、文字內容、與發文者的社群特徵後，首先，我們將圖像輸入圖像描述模型，透過CNN與Seq2Seq模型產生能描述該圖像的語句；其次，這些語句與文字內容串接，經過word embedding編碼後，以Early及Late Fusion兩種特徵融合方式，進一步結合社群特徵。最後，我們設計了多層（Multi-layer）及多單元（Multi-cell）雙向遞迴式神經網路（BRNN），並結合注意力機制賦予每個特徵不同的權重，以找出最重要的特徵並進行分類。實驗結果顯示，以Early Fusion融合所有特徵，使用基於GRU的多單元（Multi-cell）BRNN架構能達到最佳效果，F-measure達0.89，驗證了本論文所提出謠言檢測方法的有效性，未來將以更大量的資料進行實驗。

With the rapid growth of information, browsing social media on the Internet is becoming a part of people's daily lives. Social platforms give us the latest information in real time, for example, sharing personal life and commenting on social events. However, with the vigorous development of social platforms, lots of rumors and fake messages are appearing on the Internet. Most of the social platforms use manual reporting or statistics to distinguish rumors, which are very inefficient. In this paper, we propose a multimodal feature fusion approach to rumor detection by combining image captioning model with deep attention networks. First, for images extracted from tweets, we apply Image Caption model to generate captions by Convolutional Neural Networks (CNNs) and Sequence-to-Sequence (Seq2Seq) model. Second, words in captions and text contents from tweets are represented as vectors by word embedding models and combined with social features in tweets with early and late fusion strategies. Finally, we design Multi-layer and Multi-cell Bi-directional Recurrent Neural Networks (BRNNs) with attention mechanism to find word dependency and learn the most important features for classification. From the experimental results, the best F-measure of 0.89 can be obtained for our proposed Multi-cell BRNN based on Gated Recurrent Units (GRUs) with attention using early fusion of all features except for user features. This shows the potential of our proposed approach to rumor detection. Further investigation is needed for data in larger scales.