Kernel Methods

RBF kernel implementation with Linear Discriminant

Introduction

In this project a face recognition task using multivariate analysis methods will be studied. The dataset (Olivetti Faces) consists of ten different images taken from 40 distinct subjects. For some subjects, the images were taken at different times, varying the lighting, facial expressions (open / closed eyes, smiling / not smiling) and facial details (glasses / no glasses). All the images were taken against a dark homogeneous background with the subjects in an upright, frontal position (with tolerance for some side movement).

The next code includes the lines to download this data set and create the training and test data partitions.

Evaluating the performance of 5 different schemes for classification (the face recognition task is a classification task where each subject forms an output class):

1. Training the detectors

The above schemes will be trained and for each scheme the best hyperparameters will be defined with cross-validation. In particular these are the hyperparameters for each scheme:

First, the number of instances of each class has been counted in the training. As it could be seen on the below results after counting each class in Y_train, the majority class is 31 with 10 observations and the minority class is 32 and 15 with only 4 observations. Therefore, it is important to keep this balance while separating the data into folds in cross validation.

In the part below, a version of k-fold cross-validation is used which preserves the imbalanced class distribution in each fold. It is called stratified k-fold cross-validation and will enforce the class distribution in each split of the data to match the distribution in the complete training dataset. It means that it will be obtained at least one instance of each class in each fold. This stratified k fold is in the sklearn.model_selection.

Indeed, it can be seen that there are observations for each class in each fold.

1. SVM

Here, a SVM model will be fit by tuning the hyper-parameters with grid search. To make the processes easier, a pipeline is created which contains the scaler and svc process, respectively. For implying SVM, it is important to scale the data. While calling pipeline.fit, it first fit features and transform it, then fit the classifier on the transformed features sequencely.

As seen on the above results, in total 12 different models were tunned, 8 of them using rbf kernel and 4 of them using linear kernel. For both models, different C values have been tried and only for the rbf kernel different gamma values have been tried.

As a scoring criteria in cross validation, f1 score with macro average type has been used. macro has been used, because it calculates metrics for each label, and find their unweighted mean. As the data has imbalance labels, this method does not take label imbalance into account.

As a result of grid search hyper-parameter tuning, the best parameters has been obtained which are {'svcC': 10, 'svcgamma': 0.0001, 'svc__kernel': 'rbf'}, and this model's f1_macro validation score has been found 0.9308.

2. PCA and SVM

On the below part, the PCA is fit on X_train with .99, which indicates that the minimum number of principal components 99% of the variance is retained.

On the above output, the evaluation of the PCA components by the variances can be seen, and the optimal number of components that PCA choose after fitting the model using pca.n_components_.

Model Fitting

The pcastep must come right after the scaler step because it is necessary to normalize data before performing PCA.

In the param_grid, different number of pca components is defined to find the best one, and the parameters for the SVM model are the same as in the previous part.

As a result of grid search hyper-parameter tuning, the best parameters are found: {'pcan_components': 85, 'svcC': 100, 'svc__kernel': 'linear'}, and this model's f1_macro validation score has been found 0.9308. The optimal number of components has been found 85.

3. LDA

As a result of the cross validation, this model's f1_macro validation score has been found 0.9339. According to the scores, the 1st and the 3rd folds had the same validation score and the least score has been found in the 2nd fold.

4. KPCA and SVM

In this part, similar to the PCA+SVM implementation, a new step is added in the pipeline as kpca between scaler and svc steps.

In the param_grid, different number of kpca components will be tried to find the best one, and tried the same parameters for the SVM model as in the previous part.

As a result of grid search hyper-parameter tuning, the best parameters are {'kpcakernel': 'linear', 'kpcan_components': 75, 'svc__C': 10}, and this model's f1_macro validation score has been found 0.9155. The optimal number of components has been found 75 for kernal pca.

5. KDA

In this part, as indicated in the train detectors part, the predefined RBFKernelFisherDiscriminant will be used to do the Kernal discrimant analysis.

The custom kernel has a parameter gamma to tune, therefore,the model will be fit with different gamma parameters for the kernel in order to obtain the best estimator.

As a result of grid search hyper-parameter tuning,the best parameter 1e-06 for the gamma of the custom kernel. Moreover, this model's f1_macro validation score has been found 0.9483, which is the highest found validation score among the tuned models.

2. Evaluation on the test set

In this part, the winner estimators after grid search will be assessed in the test set to look for the accuracy of the chosen models.

In the below chunk, all the winning models will be iterated and collected the accuracy score. As a last step, all the scores will be written into a list in order to create a results data frame later.

Creating the result data frame with the test classification accuracies

Discussion

The accuracy of both models, LDA and KDA is the same which is 97%, the highest classification accuracy result among all the studied models. This fact points out that there is a similarity between these models. This similarity is not suprising because both KDA and LDA are reducing the dimension subspace in order to enhance the separability of the classes, and in both methods, a linear classification is done.

After these models, PCA+SVM, SVM and KPCA+SVM performed better, respectively. In general, the common thing in these three models is that SVM has been used as a classifier. However, the hyper-parameters are varying in each model's best estimator. For instance, PCA+SVM and KPCA+SVM models' best estimators use a linear kernel SVM, and SVM uses a rbf kernel. All in all, there is a slight difference in the performance of these 3 models. This is due to the iml¡plementation of dimension reduction techniques. After comparing the accuracies, The model that PCA used performed sligh better.

3. Impact of the number of components in the reduced dimension space

In this part,the reduced dimension will be obtained while classifying the test set. As in a pipeline, first steps are for transforming data and the last step is classification, the dimension reduction step used in the pipeline will be gotten by using the function named_steps. For example, for PCA+SVM model, PCA step is the step where dimension reduction takes place.

As in grid search part, for PCA and KernelPCA, the optimal number of components has been chosen, then the model is fit and transformed the test set with these chosen components. For the LDA model, the estimator has a transform function so the reduced dimension could be gotten by using this function. Also, for the custom defined kernel(kda) class, there is a function to transform the data. Thefore, this function will be called to see the reduced dimension space while classifying the images.

According to the results obtained above, as there are 4096 features in the data, no feature reduction occured in SVM. And for the other models, the reduced dimensions could be seen.

Putting the reduced dimension space information into a column in results_df.

On the above table, the reduced dimension spaces in each sheme can be seen. It is apparent that both LDA and KDA reduced the dimension from 4096 to 39. KPCA and PCA reduced to 75 and 85, respectively. As in SVM model no reduction method implied while fitting, the dimension is the same as the original size.

The evolution of the cross validation as the number of principal components increases

In the part below, the PCA spectrum can be seen while doing cross validation.

In the first visual, the evaluation of the variance ratio while the number of components increases can be seen while train fitting with PCA, and in the second visual, the classification accuracy can be seen in the validation set while the number of components increases. But these accuracies are the best classifier result for each number of components.

The evolution of the classification test error as the number of principal components increases

In the part below, the classification test errors for each tried component can be seen.

Before during the hyper-parameter tunning the pca optimal components was 85. On the above visual, it is clear that the classification test error has the least value with 35, 45 and 55 components. With all these number of components, the same test error is achieved. It is apparent that the chosen number of component (85) does not have the least test classification error.

The evolution of the cross validation as the number of components increases

In the part below, we can see the Kernel PCA spectrum while doing cross validation.

The evolution of the classification test error as the number of components increases

In the part below, we are getting the classification test errors for each tried component.

Before during the hyper-parameter tunning, 75 was found as a kpca optimal nuber of components. On the above visual, it is clear that there is a drop in the component 35, which means the classification test error has the least value. Therefore the chosen optimal component for kernel pca does not have the least error rate.

Discussion about the effective number of components that should be enough to train classifiers reasonably accurate

After evaluating the models by just changing the number of components in both pca and kpca, it is clear that the model with pca components 35, 45 and 55 the least test error is achieved, even though in cross validation the chosen number of components was 85. Therefore, in this case, these components with the least error rate could be used while training the classifiers. Moreover, for the model with kpca, instead of 75 components, 35 components would be enough to train.

4. Performance with the test set contaminated with noise

Evaluating the accuracy of the already trained detectors with noisy set

Discussion about the different robustness of each method as the noise level increases:

After adding noise to the test set, it is apparent that PCA+SVM, KPCA+SVM and KDA models are more robust with respect to the other models, although the test classification accuracy has decreased a bit for PCA+SVM and KDA compared to the test accuracies on the test set without noise. However, the test accuracy has not changed for the model with KPCA+SVM. On the other hand, the less robust model is SVM, because the accuracy on the noisy test set has sharply decreased from 93% to 2%. Moreover, LDA has not either performed very well on noisy data, as its accuracy has dropped from 97% to 55%.

5.

The KDA and the LDA provide with a reduced subspace that aims at maximizing the separability of the classes, and a linear classification is performed in such space. In this part, these reduced subspaces will be combained with a SVM as classifier. (using the KDA and LDA to transform the data and then learning a SVM with the transformed data)

KDA+SVM

Model Fitting

In both methods when adding a SVM classifier, the validation score of the model with best hyperparameters has improved with respect to the previous scores achieved by these methods. On the other hand, the accuracy on the test sets has not experienced such significant improvement. However, the detailed classification reports shows that the classification scores has improved for some classes. For instance, for both LDA+SVM and KDA+SVM models, the accuracy for classes 0 and 15 has improved. Also for LDA with SVM, the weighted average classification accuracy has improved from 0.97 to 0.98.