Reliable and Cost-Effective PoS-Tagging

Reliable and Cost-Effective PoS-Tagging

In order to achieve fast and high quality Part-of-speech (PoS) tagging, algorithms should be high accuracy and require less manually proofreading. To evaluate a tagging system, we proposed a new criterion of reliability, which is a kind of cost-effective criterion, instead of the conventional criterion of accuracy. The most cost-effective tagging algorithm is judged according to amount of manual editing and achieved final accuracy. The reliability of a tag-ging algorithm is defined to be the estimated best accuracy of the tagging under a fixed amount of proofreading. We compared the tagging accuracies and reliabilities among different tagging algorithms, such as Markov bi-gram model, Bayesian classifier, and context-rule classifier. According to our experiments, for the best cost-effective tagging algorithm, in average, 20% of sam-ples of ambivalence words need to be rechecked to achieve an estimated final accuracy of 99%. The tradeoffs between amount of proofreading and final accuracy for different algo- rithms are also compared. It concludes that an algorithm with highest accuracy may not always be the most reliable algorithm.