STATISTICAL METHODS IN NATURAL LANGUAGE PROCESSING

Abstract

This capstone project explores the application of statistical method ologies to two distinct natural language processing (NLP) tasks: machine translation between Ukrainian and Russian languages and the classifica tion of comments for hate speech detection. The study shows that the strategic integration of statistical approaches can improve performance of the machine translation and text classification problems. The imple mentation of linear regression with an added orthogonal constraint on weight vectors has resulted in higher precision scores. For the classifi cation of hate speech within textual comments, logistic regression with TF-IDF features was identified as the the most effective model in terms of AUC-ROC metric.