Installation and Loading required packages

Clasifying documents using topic models

In this notebook, the 20Newsgroup datasets will be used. It contains 20,000 documents with messages sent to different distribution lists. The dataset is available through Sklearn:

The documents could be accessed as follows:

The documents are classified in 20 groups:

In this notebook I will analyze the corpus from a topic modelling point of view and classify the documents:

1) Merging labels according to the similar groups.

2) Spliting the data in train (70%), validation (10%) and test (20%). Defining the dictionary using gensim and filter too-frequent tokens. Constructing the BoW representation of the model.

4) Building an LDA model for on the training set. Validating the number of topics, studying the consistency of the topics in the validation set. For the selected number of topics, studying their interpretability by analyzing the most likely words per topic.

5) Training a logistic regression and KNN classifier to classify each document using its LDA representation. Representing the confusion matrix in test and determining which categories are more problematic.

7) Comparing the results with a classifier that uses the TF-IDF representation.

1) Merging labels according to the following table

In this part, the labels will be replaced to merge with new label values.

On the below dataframe, the new target names and labels could be seen:

To check if everything is right, it could be seen that after replacing the labels, the number of documents is the same (18846)

After replacing with specific values, using LabelEncoder, to make the labels again from 0 to 5.

As seen on the above pie chart, the classes are imbalanced. The least class is misc news (5.2%), on the other hand, computers (26%), rec news (21.1%) and science news(21%) are more than others. Moreover, politics and religion news ratio are more or less similar with 13.9% and 12.9%, respectively. Therefore, the least class is misc news and the highest frequent class is computer news.

2) Split data in train (70%), validation (10%) and test (20%). Defining the dictionary using gensim and filter too-frequent tokens. Constructing the BoW representation of the model:

Before splitting the data into train, validation and test, text preprocessing will be done. First, to speed up the preprocessing steps, the first characters per text will be kept. To define the limit character to keep, the average character in all the texts will be checked, and also the texts that have more than 5000 characters will be counted. Because it would be advisable keeping 5000 characters per text which is enough.

On the above results, it can be seen that the average character is around 1170, and the longest text has 158791 characters. The number of texts that have more than 5000 characters are 546. As there are 18846 texts, it is not that much. Therefore, these longer texts could decrease the speed of processing. It is better to limit the number of characters.

In this part, for pre-processing, the spacy package will be used. As the texts are in English, en_core_web_md will be uploaded. Respectively, first the stopping words, punctuation marks and tokens with non-alphabetic characters will be removed.

Splitting the data into train, validation and test sets:

Here, while splitting the data, first the data will be split into train and test with the ratio 80-20. Then to validate the number of topics in the following part, the training sample will be split into training and validation with the ratio 70-10. However, in the end, the model will be trained with 80% of the data and test the model with 20% of the data.

Dictionary using gensim and BoW representation of the model:

The dictionary on the training set.

On the above, the 10 too-frequent tokens could be seen.

On the below, before building an LDA model on the training set, the corpus_bow for the training set will be created:

3) Building an LDA model for on the training set

Building an LDA model on the training set:

LDA is an unsupervised technique. LDA’s approach to topic modeling is that it considers each document as a collection of topics in a certain proportion. And each topic as a collection of keywords. Once in the algorithm, the number of topics is provided, all it does to rearrange the topics distribution within the documents and keywords distribution within the topics to obtain a good composition of topic-keywords distribution.

While I get the LDA base model, there are some parameters to be defined. As I have defined before the corpus and dictionary(id2word), I need to provide the number of topics (num_topics) as well. I set to 6 topics, however, I am going to evaluate the optimal number of topics in the following parts. chunksize controls how many documents are processed at a time in the training algorithm. Increasing chunksize will speed up training. passes controls how often I train the model on the entire corpus. It defines the number of epochs. I set to 10.

Calculating the top 20 words per topic with their probabilities

Visualizing the topic with the top 20 words

Consistency of the topics in the validation set:

First, I need to define the corpus of train and validation sets. Then i am going to use these corpuses

The c_uci, u_mass and c_v topic coherences capture the optimal number of topics by giving the coherence scores. These metrics are giving the interpretability of the topics found. So the higher is the better.

The coherence score(c_v) seems to increasing with the number of topics. As seen on the plot that the first best score is in 6 and 7 and also 8.

On the other hand, as the UMASS score is negative, it makes sense to choose the optimal number of topics with the value of u_mass close to 0. It can be interpretted that at the beginning both training and validation lines are close to 0, however, the number of topics are really less. Therefore, according to the graph, 7 would be the optimal one.

According to the UCI score, It could be seen that the first pick for training is in 6, and then 8. However, when the number of topics increases, the validation line score is going down, I should also consider that. Therefore, it may make sense to consider 8 according to this measure.

According to the Vector NPMI score, It could be seen that when the number of topic is 6, the training has the first high score, and later it increases and then at 8, it is highest. For validation, 8 is higher also. Therefore, it may make sense to consider topic number 8.

Lastly, Perplexity captures how model reacts of unseen new data. The larger the value the better the fit. In other words, how well does the model represent or reproduce the statistics of the validation data. As seen on the graph that the training is increasing constantly and validation goes down with the increase in the number of topics. However, some recent studies shows that perplexity and human judgment are often not correlated, and even sometimes slightly anti-correlated.1

Interpretability of chosen number of topics

After seeing and analyzing the results, I have decided to choose 8 as the number of topics.

When I look at the keywords of each topic, I can see that the topic 0 is related with religion news class. (words such as god, jesus, church, christ etc.) The topic 1 seems like it is related with the politics news. I can see the words government, state, public etc. Moreover, Topic 2 is related with sport, it means that this topic is related with the rec news group. Topic 3 is related with the computer news because of the words driver, disk, run, system, windows, software etc. Moreover, in topic 4, there are words work, time, people, go, day, year etc. Topic 5 has words car, bike, engine, power, model etc. Topic 6 has the most frequent word people and think. Also there are words jews, israel, fact, right etc. This could be related with politics. Topic 7 may be related with the science news group because of the words article, information, space, book etc.

Getting the traning and testing documents from the topics

As I have analyzed several topic models and found the number of optimal topics which is 8, then in this part, I am going to get the ratio of topics per document and create training and testing documents to imply supervised ML algorithms in the following section.

4) Training a logistic regression and KNN classifier to classify each document using its LDA representation

Logistic Regression Classifier

First, I am going to fit a Logistic regression classifier. While doing model fitting, I am going to try some hyper parameters in order to achieve the best score.

It is true that the overall classification's accuracy is not super high which is found 0.66. However, as in the study I have different kind of categories with different number of observations, I should look at each class individually. As I have checked before the proportion of each class, it is known that the least class was the category 3 and the largest class was category 0.

The most problematic category is mainly 3. After that category, the other problematic is category 2. As seen that f1 score of class 3 is 0.37 and class 2 is 0.53. The model classifies the category-3 news less than 0.50, which is the model has many misclassifications for this category. The false classifications are more than true ones. For example, the model have classified 70 category-3 news as computer news and 39 as rec news mistakely. Therefore, it could be interpretted that model makes more mistakes distinguishing category-0 news with category-3 news. It could be seen that this in the confusion matrix. The number of observations of category-3 news are the least. We can see that model needs more text and more specific words from that category to be able to distinguish more. Moreover, even though after category-3 news, the less observations are in category-5 news, the classification accuracy is quite high. It means that I have the significant words to define that topic in the model.

The best score has been found for the category 0 (computer news), category 1(rec news), and category 5(religion news) with the f1 scores 0.75, 0.70 and 0.70, respectively.

KNN

Normalization

Before using KNN, normally I standardize the data. However, in this case, this step is different. the KNN libraries of sklearn with BOW or TF-IDF representations cannot be used because they only allow to use the Euclidean distance. When I normalize the BoW corpus by row, then it is propotional to cosine distance. In this case, using the Euclidean distance will give the same results as the cosine distance.

After normalization, on the above part, I am going to fit a K Neighbors classifier. Again, while doing model fitting, I am going to try some hyper parameters in order to improve the model performance.

The overall accuracy has improved from 0.66 to 0.67 compared to Logistic regression. According to the results obtained, the most problematic categories are 2 and 3. In KNN as logistic regression model, the worst f1 score has been found for category-3 news. Classification score for category-2 is very similar for both models which is 0.54 for KNN and 0.53 for Logistic regression model. Apart from that, category-3 news classification has improved. True classifications has increased. With KNN, I could get 0.42 accuracy. Therefore, model performance for distinguishing the type 3 has increased compared to logistic regression model, however, it is still bad. Moreover, The best f1 scores has been found for the label 0,label 1, and label 5 with the scores 0.74, 0.71 and 0.70, respectively. In category-0 computer news, the number of true classifications has increased from 767 to 776.

5) Comparing the results with a classifier that uses the TF-IDF representation

TF-IDF Representation

The goal here is TF counts the word frequency in all documents, while IDF counts how many documents that word is found in. So, if TF is small and IDF is large, it means that word has many frequencies in general, but it has been found in certain documents. With this method I can get more specific words.

Logistic Regression

After implementing a Logistic regression classifier with TF-IDF representation, it is clear that the overall performance has increased significantly for all the classes. Still, the category-0 news about computers are classified the best with 0.85 f1 score which was 0.75 before. The category 1 news also classified with a high score which is 0.80. This class performance has increased from 0.70. Moreover, before we saw that categories 2 and 3 were the most problematic classes with 0.53 and 0.37 scores. In this case their performance increased to 0.75 and 0.76, respectively! Even though we have the least observation texts for the category 3, we got a high accuracy. With the TF-IDF representation, the model could distinguished much more the category-3 news.

According to the confusion matrix, 60 category-0 news have been classified as category 1 and 55 of them also classified as category-2 mistakely. This makes sense because for example category-2 is about science, and these news could be related with the computer news. However, here it is important that we could increase the true classifications with the TF-IDF representation. Because, in the previous model, we had more misclassifications on the category-2 for computer type news. Moreover, in the matrix, we see that wrong classifications for category-1 are more on the category-2 news, wrong classifications for category-2 are more on the category-0 and for category-3 are more on the category-0 as well and so on..

KNN

First, I used the normalization function I have defined in the previous part, and I normalize also the tf-idf corpus for train and test before implementing KNN.

According to the detailed classification report above, the accuracy of KNN when TF-IDF used is worse than the KNN when BoW used. It indicates that BoW representation is much more suitable weighting scheme for this analysis using KNN.

It can be seen that the overall accuracy has dropped sharply from 0.67 to 0.45. As there are significant differences between precision and recall scores, it is better to interpret them separately. For instance, when we look at the confusion matrix, 630 category-0 classes that are related with computers have been missclassified as science news. Compared to the previous KNN model, the false classifiactions are way higher than the true classifications. Thats why the recall score is really less, 0.32. Moreover, the precision score is very high which is 0.78. This indicates that the proportion of correctly classified computer news class in the all predicted values are higher. Moreover, the recall value for the category-1 news is also really less and we can see that the misclassifications are higher compared to the KNN with BoW. Additionally, when we look at the category-2 news which are related with science, its true classifications has increased compared to the KNN with BoW. We can see that this class recall score is 0.92.

SGDClassifier

Extra: In this part, I wanted to implement also a SGD classifier with hinge loss in order to increase the performance. While doing that I created a pipeline that uses CountVectorizer and then TfidTransformer. In the Count vectorizer, actually I can do text preprocessing, tokenizing and filtering of stopwords. They are all included in it, and it builds a dictionary of features and transforms documents to feature vectors. It counts the occurences of the words. However, after counting, for example, words in longer documents related to the same topic may have a higher average count value than words in shorter documents. Actually, to avoid this, in the tf-idf transformer step, first the number of occurrences of each word are divided by the total number of words in the document, and then the weights of the words in the sentence are reduced, which are less informative than words in many documents in the sentence. After these two transformation processes in the pipeline, then SGD classifier with hinge loss and l2 type penalty is used in grid search with different parameters to achieve the best score. Apart from the classifier hyper params, I also have checked the different ngram range for vectorizer and with and without tf-idf transformer.

As seen on the above result, the best model has been found with the tf-idf representation. When I look at the scores and true classifications in the confusion matrix, for all the classes, the missclassification rate has been increased significantly. The overall accuracy of the model is the best compared to other tried models.