Log In

Text Classification with Bag of Words - Natural Language Processing

Open In Colab

"Natural language processing (NLP) is a subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data." - Wikipedia

Bag of Words: The bag-of-words (BOW) model is a representation that turns arbitrary text into fixed-length vectors by counting how many times each word appears.

Outline:

  1. Download and explore a real-world dataset
  2. Apply text preprocessing techniques
  3. Implement the bag of words model
  4. Train ML models for text classification
  5. Make predictions and submit to Kaggle

Dataset: https://www.kaggle.com/c/quora-insincere-questions-classification

Download and Explore the Data

Outline:

  1. Download the dataset from Kaggle to Colab
  2. Explore the data using Pandas
  3. Create a small working sample

Download the Data to Colab

Upload the kaggle.json to Colab. Get it here: https://www.kaggle.com/docs/api#authentication

!ls .
kaggle.json sample_data 
import os
IS_KAGGLE = 'KAGGLE_KERNEL_RUN_TYPE' in os.environ
if IS_KAGGLE:
    data_dir = '../input/quora-insincere-questions-classification'
    train_fname = data_dir + '/train.csv'
    test_fname = data_dir + '/test.csv'
    sample_fname = data_dir + '/sample_submission.csv'
else:
    os.environ['KAGGLE_CONFIG_DIR'] = '.'
    !kaggle competitions download -c quora-insincere-questions-classification -f train.csv -p data
    !kaggle competitions download -c quora-insincere-questions-classification -f test.csv -p data
    !kaggle competitions download -c quora-insincere-questions-classification -f sample_submission.csv -p data
    train_fname = 'data/train.csv.zip'
    test_fname = 'data/test.csv.zip'
    sample_fname = 'data/sample_submission.csv.zip'
Warning: Your Kaggle API key is readable by other users on this system! To fix this, you can run 'chmod 600 ./kaggle.json' train.csv.zip: Skipping, found more recently modified local copy (use --force to force download) Warning: Your Kaggle API key is readable by other users on this system! To fix this, you can run 'chmod 600 ./kaggle.json' test.csv.zip: Skipping, found more recently modified local copy (use --force to force download) Warning: Your Kaggle API key is readable by other users on this system! To fix this, you can run 'chmod 600 ./kaggle.json' sample_submission.csv.zip: Skipping, found more recently modified local copy (use --force to force download)

Explore the Data using Pandas

import pandas as pd
raw_df = pd.read_csv(train_fname)
raw_df
sincere_df = raw_df[raw_df.target == 0]
sincere_df.question_text.values[:10]
array(['How did Quebec nationalists see their province as a nation in the 1960s?',
       'Do you have an adopted dog, how would you encourage people to adopt and not shop?',
       'Why does velocity affect time? Does velocity affect space geometry?',
       'How did Otto von Guericke used the Magdeburg hemispheres?',
       'Can I convert montra helicon D to a mountain bike by just changing the tyres?',
       'Is Gaza slowly becoming Auschwitz, Dachau or Treblinka for Palestinians?',
       'Why does Quora automatically ban conservative opinions when reported, but does not do the same for liberal views?',
       'Is it crazy if I wash or wipe my groceries off? Germs are everywhere.',
       'Is there such a thing as dressing moderately, and if so, how is that different than dressing modestly?',
       'Is it just me or have you ever been in this phase wherein you became ignorant to the people you once loved, completely disregarding their feelings/lives so you get to have something go your way and feel temporarily at ease. How did things change?'],
      dtype=object)
insincere_df = raw_df[raw_df.target == 1]
insincere_df.question_text.values[:10]
array(['Has the United States become the largest dictatorship in the world?',
       'Which babies are more sweeter to their parents? Dark skin babies or light skin babies?',
       "If blacks support school choice and mandatory sentencing for criminals why don't they vote Republican?",
       'I am gay boy and I love my cousin (boy). He is sexy, but I dont know what to do. He is hot, and I want to see his di**. What should I do?',
       'Which races have the smallest penis?',
       'Why do females find penises ugly?',
       'How do I marry an American woman for a Green Card? How much do they charge?',
       "Why do Europeans say they're the superior race, when in fact it took them over 2,000 years until mid 19th century to surpass China's largest economy?",
       'Did Julius Caesar bring a tyrannosaurus rex on his campaigns to frighten the Celts into submission?',
       "In what manner has Republican backing of 'states rights' been hypocritical and what ways have they actually restricted the ability of states to make their own laws?"],
      dtype=object)
raw_df.target.value_counts(normalize=True)
0    0.93813
1    0.06187
Name: target, dtype: float64
raw_df.target.value_counts(normalize=True).plot(kind='bar')
<matplotlib.axes._subplots.AxesSubplot at 0x7f5f99a4fe90>
Notebook output
test_df = pd.read_csv(test_fname)
test_df
sub_df = pd.read_csv(sample_fname)
sub_df
sub_df.prediction.value_counts()
0    375806
Name: prediction, dtype: int64

Create a Working Sample

if IS_KAGGLE:
    SAMPLE_SIZE = len(raw_df)
else:
    SAMPLE_SIZE = 100_000
sample_df = raw_df.sample(SAMPLE_SIZE, random_state=42)
sample_df

Text Preprocessing Techniques

Outline:

  1. Understand the bag of words model
  2. Tokenization
  3. Stop word removal
  4. Stemming

Bag of Words Intuition

  1. Create a list of all the words across all the text documents
  2. You convert each document into vector counts of each word

Limitations:

  1. There may be too many words in the dataset
  2. Some words may occur too frequently
  3. Some words may occur very rarely or only once
  4. A single word may have many forms (go, gone, going or bird vs. birds)
q0 = sincere_df.question_text.values[1]
q0
'Do you have an adopted dog, how would you encourage people to adopt and not shop?'
q1 = raw_df[raw_df.target == 1].question_text.values[0]
q1
'Has the United States become the largest dictatorship in the world?'

Tokenization

splitting a document into words and separators

import nltk
from nltk.tokenize import word_tokenize
nltk.download('punkt')
[nltk_data] Downloading package punkt to /root/nltk_data... [nltk_data] Unzipping tokenizers/punkt.zip. 
True
q0
'Do you have an adopted dog, how would you encourage people to adopt and not shop?'
word_tokenize(q0)
['Do',
 'you',
 'have',
 'an',
 'adopted',
 'dog',
 ',',
 'how',
 'would',
 'you',
 'encourage',
 'people',
 'to',
 'adopt',
 'and',
 'not',
 'shop',
 '?']
word_tokenize(' this is (something) with, a lot of, punctuation;')
['this',
 'is',
 '(',
 'something',
 ')',
 'with',
 ',',
 'a',
 'lot',
 'of',
 ',',
 'punctuation',
 ';']
q1
'Has the United States become the largest dictatorship in the world?'
word_tokenize(q1)
['Has',
 'the',
 'United',
 'States',
 'become',
 'the',
 'largest',
 'dictatorship',
 'in',
 'the',
 'world',
 '?']
q0_tok = word_tokenize(q0)
q1_tok = word_tokenize(q1)

Stop Word Removal

Removing commonly occuring words

q1_tok
['Has',
 'the',
 'United',
 'States',
 'become',
 'the',
 'largest',
 'dictatorship',
 'in',
 'the',
 'world',
 '?']
from nltk.corpus import stopwords
nltk.download('stopwords')
[nltk_data] Downloading package stopwords to /root/nltk_data... [nltk_data] Unzipping corpora/stopwords.zip. 
True
english_stopwords = stopwords.words('english')
", ".join(english_stopwords)
"i, me, my, myself, we, our, ours, ourselves, you, you're, you've, you'll, you'd, your, yours, yourself, yourselves, he, him, his, himself, she, she's, her, hers, herself, it, it's, its, itself, they, them, their, theirs, themselves, what, which, who, whom, this, that, that'll, these, those, am, is, are, was, were, be, been, being, have, has, had, having, do, does, did, doing, a, an, the, and, but, if, or, because, as, until, while, of, at, by, for, with, about, against, between, into, through, during, before, after, above, below, to, from, up, down, in, out, on, off, over, under, again, further, then, once, here, there, when, where, why, how, all, any, both, each, few, more, most, other, some, such, no, nor, not, only, own, same, so, than, too, very, s, t, can, will, just, don, don't, should, should've, now, d, ll, m, o, re, ve, y, ain, aren, aren't, couldn, couldn't, didn, didn't, doesn, doesn't, hadn, hadn't, hasn, hasn't, haven, haven't, isn, isn't, ma, mightn, mightn't, mustn, mustn't, needn, needn't, shan, shan't, shouldn, shouldn't, wasn, wasn't, weren, weren't, won, won't, wouldn, wouldn't"
def remove_stopwords(tokens):
    return [word for word in tokens if word.lower() not in english_stopwords]
q0_tok
['Do',
 'you',
 'have',
 'an',
 'adopted',
 'dog',
 ',',
 'how',
 'would',
 'you',
 'encourage',
 'people',
 'to',
 'adopt',
 'and',
 'not',
 'shop',
 '?']
q0_stp = remove_stopwords(q0_tok)
q0_stp
['adopted', 'dog', ',', 'would', 'encourage', 'people', 'adopt', 'shop', '?']
q1_stp = remove_stopwords(q1_tok)
q1_tok
['Has',
 'the',
 'United',
 'States',
 'become',
 'the',
 'largest',
 'dictatorship',
 'in',
 'the',
 'world',
 '?']
q1_stp
['United', 'States', 'become', 'largest', 'dictatorship', 'world', '?']

Stemming

"go", "gone", "going" -> "go" "birds", "bird" -> "bird"

from nltk.stem.snowball import SnowballStemmer
stemmer = SnowballStemmer(language='english')
stemmer.stem('going')
'go'
stemmer.stem('supposedly')
'suppos'
q0_stm = [stemmer.stem(word) for word in q0_stp]
q0_stp
['adopted', 'dog', ',', 'would', 'encourage', 'people', 'adopt', 'shop', '?']
q0_stm
['adopt', 'dog', ',', 'would', 'encourag', 'peopl', 'adopt', 'shop', '?']
q1_stm = [stemmer.stem(word) for word in q1_stp]
q1_stp
['United', 'States', 'become', 'largest', 'dictatorship', 'world', '?']
q1_stm
['unit', 'state', 'becom', 'largest', 'dictatorship', 'world', '?']

Lemmatization

"love" -> "love" "loving" -> "love" "lovable" -> "love"

Implement Bag of Words

Outline:

  1. Create a vocabulary using Count Vectorizer
  2. Transform text to vectors using Count Vectorizer
  3. Configure text preprocessing in Count Vectorizer

Create a Vocabulary

sample_df
small_df = sample_df[:5]
small_df
small_df.question_text.values
array(['What is the most effective classroom management skill/technique to create a good learning environment?',
       'Can I study abroad after 10th class from Bangladesh?',
       'How can I make friends as a college junior?',
       'How do I download free APK Minecraft: Pocket Edition for iOS (iPhone)?',
       'Like Kuvera, is "Groww" also a free online investment platform where I can invest in direct mutual funds?'],
      dtype=object)
from sklearn.feature_extraction.text import CountVectorizer
small_vect = CountVectorizer()
small_vect.fit(small_df.question_text)
CountVectorizer()
small_vect.get_feature_names_out()
array(['10th', 'abroad', 'after', 'also', 'apk', 'as', 'bangladesh',
       'can', 'class', 'classroom', 'college', 'create', 'direct', 'do',
       'download', 'edition', 'effective', 'environment', 'for', 'free',
       'friends', 'from', 'funds', 'good', 'groww', 'how', 'in', 'invest',
       'investment', 'ios', 'iphone', 'is', 'junior', 'kuvera',
       'learning', 'like', 'make', 'management', 'minecraft', 'most',
       'mutual', 'online', 'platform', 'pocket', 'skill', 'study',
       'technique', 'the', 'to', 'what', 'where'], dtype=object)

Transform documents into Vectors

vectors = small_vect.transform(small_df.question_text)
vectors
<5x51 sparse matrix of type '<class 'numpy.int64'>'
	with 56 stored elements in Compressed Sparse Row format>
vectors.shape
(5, 51)
small_df.question_text.values[0]
'What is the most effective classroom management skill/technique to create a good learning environment?'
vectors[0].toarray()
array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0,
        0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0,
        1, 0, 1, 1, 1, 1, 0]])
vectors.toarray()
array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0,
        0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0,
        1, 0, 1, 1, 1, 1, 0],
       [1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 1, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,
        0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0,
        0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1,
        0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0,
        1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0,
        0, 0, 0, 0, 0, 0, 1]])

Configure Count Vectorizer Parameters

stemmer = SnowballStemmer(language='english')
def tokenize(text):
    return [stemmer.stem(word) for word in word_tokenize(text)]
tokenize('What is the really (dealing) here?')
['what', 'is', 'the', 'realli', '(', 'deal', ')', 'here', '?']
vectorizer = CountVectorizer(lowercase=True, 
                             tokenizer=tokenize,
                             stop_words=english_stopwords,
                             max_features=1000)
%%time
vectorizer.fit(sample_df.question_text)
/usr/local/lib/python3.7/dist-packages/sklearn/feature_extraction/text.py:517: UserWarning: The parameter 'token_pattern' will not be used since 'tokenizer' is not None' "The parameter 'token_pattern' will not be used" /usr/local/lib/python3.7/dist-packages/sklearn/feature_extraction/text.py:401: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ["'d", "'s", 'abov', 'ani', 'becaus', 'befor', 'could', 'doe', 'dure', 'might', 'must', "n't", 'need', 'onc', 'onli', 'ourselv', 'sha', 'themselv', 'veri', 'whi', 'wo', 'would', 'yourselv'] not in stop_words. % sorted(inconsistent)
CPU times: user 32.1 s, sys: 70.3 ms, total: 32.2 s Wall time: 32.3 s 
CountVectorizer(max_features=1000,
                stop_words=['i', 'me', 'my', 'myself', 'we', 'our', 'ours',
                            'ourselves', 'you', "you're", "you've", "you'll",
                            "you'd", 'your', 'yours', 'yourself', 'yourselves',
                            'he', 'him', 'his', 'himself', 'she', "she's",
                            'her', 'hers', 'herself', 'it', "it's", 'its',
                            'itself', ...],
                tokenizer=<function tokenize at 0x7f5f81cdd7a0>)
len(vectorizer.vocabulary_)
1000
vectorizer.get_feature_names_out()[:100]
array(['!', '$', '%', '&', "'", "''", "'m", "'s", '(', ')', ',', '-', '.',
       '1', '10', '100', '12', '12th', '15', '2', '20', '2017', '2018',
       '3', '4', '5', '6', '7', '8', ':', '?', '[', ']', '``', 'abl',
       'abroad', 'abus', 'accept', 'access', 'accomplish', 'accord',
       'account', 'achiev', 'act', 'action', 'activ', 'actor', 'actual',
       'ad', 'add', 'address', 'admiss', 'adult', 'advanc', 'advantag',
       'advic', 'affect', 'africa', 'african', 'age', 'agre', 'air',
       'allow', 'almost', 'alon', 'alreadi', 'also', 'altern', 'alway',
       'amazon', 'america', 'american', 'amount', 'analysi', 'android',
       'ani', 'anim', 'anoth', 'answer', 'anxieti', 'anyon', 'anyth',
       'apart', 'app', 'appear', 'appl', 'appli', 'applic', 'approach',
       'arab', 'area', 'armi', 'around', 'art', 'asian', 'ask', 'associ',
       'atheist', 'attack', 'attend'], dtype=object)
%%time
inputs = vectorizer.transform(sample_df.question_text)
CPU times: user 30.4 s, sys: 60.9 ms, total: 30.5 s Wall time: 30.5 s 
inputs.shape
(100000, 1000)
inputs
<100000x1000 sparse matrix of type '<class 'numpy.int64'>'
	with 548298 stored elements in Compressed Sparse Row format>
sample_df.question_text.values[0]
'What is the most effective classroom management skill/technique to create a good learning environment?'
test_df
%%time
test_inputs = vectorizer.transform(test_df.question_text)
CPU times: user 1min 52s, sys: 317 ms, total: 1min 52s Wall time: 1min 52s

ML Models for Text Classification

Outline:

  • Create a training & validation set
  • Train a logistic regression model
  • Make predictions on training, validation & test data

Split into Training and Validation Set

sample_df
inputs.shape
(100000, 1000)
from sklearn.model_selection import train_test_split
train_inputs, val_inputs, train_targets, val_targets = train_test_split(inputs, sample_df.target, 
                                                                        test_size=0.3, random_state=42)
train_inputs.shape
(70000, 1000)
train_targets.shape
(70000,)
val_inputs.shape
(30000, 1000)
val_targets.shape
(30000,)

Train Logistic Regression model

from sklearn.linear_model import LogisticRegression
MAX_ITER = 1000
model = LogisticRegression(max_iter=MAX_ITER, solver='sag')
%%time
model.fit(train_inputs, train_targets)
CPU times: user 26.9 s, sys: 17 ms, total: 26.9 s Wall time: 26.8 s
/usr/local/lib/python3.7/dist-packages/sklearn/linear_model/_sag.py:354: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge ConvergenceWarning, 
LogisticRegression(max_iter=1000, solver='sag')

Make predictions using the model

train_preds = model.predict(train_inputs)
train_targets
133883     0
343925     0
1190558    0
317078     0
355610     0
          ..
262505     0
879467     0
411614     0
859707     0
683832     0
Name: target, Length: 70000, dtype: int64
train_preds
array([0, 0, 0, ..., 0, 0, 0])
pd.Series(train_preds).value_counts()
0    67957
1     2043
dtype: int64
pd.Series(train_targets).value_counts()
0    65784
1     4216
Name: target, dtype: int64
from sklearn.metrics import accuracy_score
accuracy_score(train_targets, train_preds)
0.9504428571428571
import numpy as np
accuracy_score(train_targets, np.zeros(len(train_targets)))
0.9397714285714286
from sklearn.metrics import f1_score
f1_score(train_targets, train_preds)
0.4457581083240134
f1_score(train_targets, np.zeros(len(train_targets)))
0.0
random_preds = np.random.choice((0, 1), len(train_targets))
f1_score(train_targets, random_preds)
0.10605752617135888
val_preds = model.predict(val_inputs)
accuracy_score(val_targets, val_preds)
0.9467
f1_score(val_targets, val_preds)
0.40843507214206437
sincere_df.question_text.values[:10]
array(['How did Quebec nationalists see their province as a nation in the 1960s?',
       'Do you have an adopted dog, how would you encourage people to adopt and not shop?',
       'Why does velocity affect time? Does velocity affect space geometry?',
       'How did Otto von Guericke used the Magdeburg hemispheres?',
       'Can I convert montra helicon D to a mountain bike by just changing the tyres?',
       'Is Gaza slowly becoming Auschwitz, Dachau or Treblinka for Palestinians?',
       'Why does Quora automatically ban conservative opinions when reported, but does not do the same for liberal views?',
       'Is it crazy if I wash or wipe my groceries off? Germs are everywhere.',
       'Is there such a thing as dressing moderately, and if so, how is that different than dressing modestly?',
       'Is it just me or have you ever been in this phase wherein you became ignorant to the people you once loved, completely disregarding their feelings/lives so you get to have something go your way and feel temporarily at ease. How did things change?'],
      dtype=object)
sincere_df.target.values[:10]
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
model.predict(vectorizer.transform(sincere_df.question_text.values[:10]))
array([0, 0, 0, 0, 0, 0, 1, 0, 0, 0])
insincere_df.question_text.values[:10]
array(['Has the United States become the largest dictatorship in the world?',
       'Which babies are more sweeter to their parents? Dark skin babies or light skin babies?',
       "If blacks support school choice and mandatory sentencing for criminals why don't they vote Republican?",
       'I am gay boy and I love my cousin (boy). He is sexy, but I dont know what to do. He is hot, and I want to see his di**. What should I do?',
       'Which races have the smallest penis?',
       'Why do females find penises ugly?',
       'How do I marry an American woman for a Green Card? How much do they charge?',
       "Why do Europeans say they're the superior race, when in fact it took them over 2,000 years until mid 19th century to surpass China's largest economy?",
       'Did Julius Caesar bring a tyrannosaurus rex on his campaigns to frighten the Celts into submission?',
       "In what manner has Republican backing of 'states rights' been hypocritical and what ways have they actually restricted the ability of states to make their own laws?"],
      dtype=object)
insincere_df.target.values[:10]
array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
model.predict(vectorizer.transform(insincere_df.question_text.values[:10]))
array([0, 0, 1, 1, 0, 0, 0, 1, 0, 0])

Make Predictions and Submit to Kaggle

test_df
test_inputs.shape
(375806, 1000)
test_preds = model.predict(test_inputs)
sub_df
sub_df.prediction = test_preds
sub_df.prediction.value_counts()
0    364965
1     10841
Name: prediction, dtype: int64
sub_df
sub_df.to_csv('submission.csv', index=None)
!head submission.csv
qid,prediction 0000163e3ea7c7a74cd7,0 00002bd4fb5d505b9161,0 00007756b4a147d2b0b3,0 000086e4b7e1c7146103,0 0000c4c3fbe8785a3090,0 000101884c19f3515c1a,0 00010f62537781f44a47,0 00012afbd27452239059,0 00014894849d00ba98a9,0

References