Word Level Language Identification of Code Mixing Text in Social Media using NLP

Kasthuri Shanmugalingam, Sagara Sumathipala, Chinthaka Premachandra

研究成果: Conference contribution

1 引用 (Scopus)

抜粋

Understanding social media contents has been a primary research topic since the dawn of social networking. Especially, contextual understanding of the noisy text, which is characterized by a high percentage of spelling mistakes with creative spelling, phonetic typing, wordplay, abbreviations, and Meta tags. Thus, the data processing demands a more complex system than traditional natural language processors. Also people easily mixing two or more languages together to express their thoughts in social media context. So automatic language identification at word level become as necessary part for analyzing the noisy content in social media. It would help with the automated analysis of content generated on social media. This study uses Tamil-English code-mixed data from popular social media posts and comments and provided word level language tags using Natural Language Processing (NLP) and modern Machine Learning (ML) technologies. The methodology used for this system is a novel approach implemented as machine learning classifier based on features such as Tamil Unicode characters in Roman scripts, dictionaries, double consonant, and term frequency. Different machine learning classifiers such as Naive Bayes, Logistic Regression, Support Vector Machines (SVM), Decision Trees and Random Forest used in training and testing. Among that the highest accuracy of 89.46% was obtained in SVM classifier.

元の言語English
ホスト出版物のタイトル2018 3rd International Conference on Information Technology Research, ICITR 2018
出版者Institute of Electrical and Electronics Engineers Inc.
ISBN(電子版)9781728114705
DOI
出版物ステータスPublished - 2018 12
イベント3rd International Conference on Information Technology Research, ICITR 2018 - Moratuwa, Sri Lanka
継続期間: 2018 12 52018 12 7

出版物シリーズ

名前2018 3rd International Conference on Information Technology Research, ICITR 2018

Conference

Conference3rd International Conference on Information Technology Research, ICITR 2018
Sri Lanka
Moratuwa
期間18/12/518/12/7

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Networks and Communications
  • Computer Science Applications
  • Information Systems and Management
  • Media Technology

フィンガープリント Word Level Language Identification of Code Mixing Text in Social Media using NLP' の研究トピックを掘り下げます。これらはともに一意のフィンガープリントを構成します。

  • これを引用

    Shanmugalingam, K., Sumathipala, S., & Premachandra, C. (2018). Word Level Language Identification of Code Mixing Text in Social Media using NLP. : 2018 3rd International Conference on Information Technology Research, ICITR 2018 [8736127] (2018 3rd International Conference on Information Technology Research, ICITR 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICITR.2018.8736127