• Built a multilingual text classification model to predict the probability that a comment is toxic using the data provided by Google Jigsaw.
• The data had 4,35,775 text comments in 7 different languages.
• A RNN model was used as a baseline. The BERT-Multilingual-base and XLMRoBERTa models were fine-tuned to get the best results.
• The best results were obtained using the fine-tuned XLMRoberta model. It achieved an Accuracy of 96.24% and an ROC-AUC Score of 93.92%.

Data

A single toxic comment can ruin an online discussion. Toxicity is anything that is rude, disrespectful, or likely to make someone leave the conversation. If we can identify these toxic contributions, we can create a safer and more collaborative internet.

The goal is to find the probability that a comment is toxic. This can be done by using machine learning algorithms to analyze the text of a comment and identify words or phrases that are associated with toxicity. The algorithm can then calculate the probability that a comment is toxic based on the number of toxic words or phrases it contains.

This information can then be used to flag toxic comments for review by human moderators. By identifying and removing toxic comments, we can help to create a more positive and productive online environment for everyone.

Columns in the dataset:

id - identifier within each file.
comment_text - the text of the comment to be classified.
lang - the language of the comment.
toxic - whether or not the comment is classified as toxic.

The comments are composed of multiple non-English languages and come either from Civil Comments or Wikipedia talk page edits.

The dataset can be downloaded from Kaggle.

Experiments:

RNN:

A baseline was created using the RNN model. An embedding layer of size 64 was used. The model was trained with an Adam optimizer, employing a learning rate of 0.001, over the course of 5 epochs. The RNN model exhibited a remarkable accuracy of 83.68%, showcasing its ability to effectively process sequential data. Furthermore, the model achieved an ROC-AUC Score of 55.72%, indicating its proficiency in distinguishing between positive and negative sentiment.

BERT-Multilingual-base:

To leverage the power of pre-trained transformer models, the BERT-Multilingual-base was fine-tuned on the provided dataset. With the addition of a hidden layer containing 1024 neurons, the model was primed for exceptional performance. Training was conducted using the Adam optimizer, utilizing a learning rate of 0.001 and weight decay of 1e-6, extending for 10 epochs. Impressively, the fine-tuned BERT-Multilingual-base model yielded an outstanding accuracy of 93.92%, exemplifying its capacity to grasp the nuances of multilingual sentiment analysis. Additionally, it attained an impressive ROC-AUC Score of 89.55%, substantiating its robustness in making precise predictions across diverse languages.

XLM RoBERTa:

To further elevate the accuracy and generalization of the sentiment analysis task, the XLMRoberta model was meticulously fine-tuned on the available dataset. Employing the AdamW optimizer with a learning rate of 1e-5 and weight decay of 1e-5, the model was meticulously trained over the span of 7 epochs. The XLMRoberta model showcased exceptional performance, achieving an extraordinary accuracy of 96.24%. This exemplifies its remarkable ability to capture subtle linguistic nuances and comprehend sentiment across various languages. Furthermore, the model garnered an outstanding ROC-AUC Score of 93.92%, underscoring its efficacy in discriminating between positive and negative sentiments with a high degree of confidence.

For all the models that were fine-tuned:

• A batch size of 64 was employed during the training process, ensuring efficient computation and effective utilization of computational resources.
• Binary Cross-Entropy, a commonly adopted loss function for binary classification tasks, was utilized to train the models. It enabled accurate estimation of the dissimilarity between predicted and true sentiment labels.

Results:

Amongst the various models experimented with, the fine-tuned XLMRoberta model emerged as the clear winner, boasting the highest performance scores. Therefore, it was chosen as the ultimate choice for generating the final predictions. The fine-tuned XLMRoberta model demonstrated unparalleled accuracy, reaching an impressive 96.24%. Furthermore, it achieved a remarkable ROC-AUC Score of 93.92%. These extraordinary results solidify the model's status as the top performer, underscoring its immense capability to discern sentiment across multiple languages with exceptional precision and reliability.

The results from all the models have been summarized below:

Run Locally

The code to run the project can be found here: Jigsaw - Multilingual Toxic Comment Classification Github.

1. Install required libraries:

  pip install -r requirements.txt

1. Baseline model:

  python toxic-baseline-rnn.py

1. Fine-tune models:

  python toxic-bertm-base.py
  python toxic-xlm-roberta.py

Feedback

If you have any feedback, please reach out to me.