Bilingual Parallel Sentence Extraction from Comparable Corpora

Bilingual Parallel Sentence Extraction from Comparable Corpora

This research aims to develop a parallel sentence extraction method for automatically extracting parallel sentence pairs from bilingual comparable corpora based on cross-lingual word embeddings. Our task is to effectively identify matched sentence pairs from a Chinese-English corpus with the goal of maximizing F1 score. Our method employs pre-trained, task-specific, and hybrid (a combination of pre-trained and task-specific) monolingual word embeddings to construct a cross-lingual transformation matrix respectively to transform the word embeddings between the two languages, and develops two search strategies (sequential and exhaustive) for parallel sentence extraction. Our empirical evaluation results suggest that task-specific word embeddings (directly trained from a task-relevant corpus, i.e., 25,695 Chinese and English abstracts of theses) outperforms their counterparts. With respect to the two search strategies, our evaluation results suggest that the exhaustive search strategy attains a higher recall rate; the sequential search strategy is more efficient in time. Both strategies achieve a promising performance, with an F1 score up to 60.18%.