dc.contributor.author | Kozhirbayev, Zhanibek | |
dc.contributor.author | Yessenbayev, Zhandos | |
dc.contributor.author | Makazhanov, Aibek | |
dc.date.accessioned | 2018-10-22T05:49:17Z | |
dc.date.available | 2018-10-22T05:49:17Z | |
dc.date.issued | 2018-10 | |
dc.identifier.uri | http://nur.nu.edu.kz/handle/123456789/3549 | |
dc.description.abstract | We present herein our work on language identification applied to comments left by the readers of online news sites popular in Kazakhstan. Such comments are typically written in one of the two languages spoken widely in the area (Kazakh and Russian) and sometimes - in a mixture of both. Code-switching (mixing languages) makes it desirable to identify language not only on document, but also on individual word level. We approach both tasks in a single two-step framework, performing unsupervised normalization and Nave Bayes text classification procedures successively. Moreover, we applied deep learning model based on recurrent networks with LSTM cell in order to classify text. Our results suggest improvement over the state-of-the-art for Kazakh language. | en_US |
dc.description.sponsorship | This work has been supported by Nazarbayev University research grant 129-2017/022-2017 and the Committee of Science of the Ministry of Education and Science of the Republic of Kazakhstan under the research grant AP05134272. | en_US |
dc.language.iso | en | en_US |
dc.publisher | The IEEE 12th International Conference Application of Information and Communication Technologies | en_US |
dc.rights | Attribution-NonCommercial-NoDerivs 3.0 United States | * |
dc.rights.uri | http://creativecommons.org/licenses/by-nc-nd/3.0/us/ | * |
dc.subject | language identification, code-switching, user generated content, normalization | en_US |
dc.title | Document and Word-level Language Identification for Noisy User Generated Text | en_US |
dc.type | Conference Paper | en_US |
workflow.import.source | science |
The following license files are associated with this item: