Kaggle Comptetion

Competition Link:https://www.kaggle.com/c/house-prices-advanced-regression-techniques

Reading the data and creating the data samples.

Splitting the data into train and test (2/3 for train and 1/3 for test).

3.1 Model selection

In this section we will implement several machine learning algorithms that allow us to create a model able to predict de houses' prices in the Boston dataset. In this section we will attach first to the default hyper-parameter values and then, we will perform hyper-parameter tuning in order to improve the performance of the models.

Learning Curves

The visualization of the learning curves is very important to do model selection. If the model is overfitting, the learning curve will present a gap between the training and validation scores. In this case, the complexity of the model should be reduced. Moreover, if the model is underfitting, the learning curves will converge to a low score value. In this case, hyper-parameters tuning could improve such model. \ We will plot the learning curves from each model in order to detect any of the above-mentioned situations.

We will implement 3-fold cross-validation in order to estimate the performance of the model. The Root Mean Squared Error (RMSE) will be computed for each fold of the cross-validation, and the estimated performance will be the mean of the three results.

Model1: KNN regression

The RMSE of this model is approximately 0.20. As we can see from the learning curves plot, there is no overfitting, neither underfitting, although the performance could be improved.

Model2: Decision Trees

The RMSE is even larger than in the previous model. Moreover, the learning curves reveal that this model is overfitting, as the model fits perfectly the training data, while achieving very little performance when validating new data.

Model3: Extra trees

The extra trees gives clearly better results than the simple tree regressor. In this case the RMSE is reduced and there is no overfitting.

Model4: Random Forest

Although the performance of this model is very similar to the previous one, extra trees gives still better results.

Model 5: HistGradient Regressor

So far, this is the method that gives the best model. \

The performace of the five implemented methods are gathered in the following table:

Method RMSE Exec. time (s)
KNN Regressor 0.1968 0.0480
Decision Tree Regressor 0.2282 0.0558
Random Tree Regressor 0.1555 3.0755
Extra Tree Regressor 0.1545 2.5495
HistGradient Regressor 0.1528 2.1319

In this section we will try to improve the performance of the model given by KNN. I order to do so, we implement GridSearchCV. This function finds the hyper-parameter value (among the ones that are proposed) that minimizes the RMSE. This is done by 3-fold cross-validation on the training partition.

The goal is to find the best maximum depth and the minimum samples split. Unlike, GridSearchCV, RandomizedSearchCV does not try the whole set of hyper-parameters. Instead, it chooses some values randomly and compares the performance of the resulted models.

To tune the hyper-parameters of Extra trees (again maximum depth and minimum samples split) we implement a Bayes Search. As before, not all the candidate values are tested, but in this case they are not chosen randomly either. Bayes SearchCV seeks the hyper-parameters that are more likely to improve the performance, ignoring the values that it considers are not worthy to try.

3.1.5 Hyperparameter Tuning Gradient Descent

In this section we will use the Optuna package. Optuna also implements by default a Bayes search to do hyper-parameter tuning. However, the coding is different from what we have seen so far. It is necessary to define the objective function, that is, the function we want to optimize. In each iteration, this function is called and evaluated with the new set of suggested hyper-parameters.

3.1.6 Optuna research

Pruning algorithms with OPTUNA

Pruning algorithms allow terminating the learning process prematurely based on the result of previous trials. Pruning mechanism in general works in two phases: \ (1) It periodically monitors the intermediate objective values. \ (2) It terminates the trial that does not meet the predefined condition. \

In addition to the median stopping rule, which is the pruner that has been used below, Optuna also supports algorithms such as Successive Halving and Hyperband , but the basic ideas remain largely the same.

The median stopping rule is based on the following idea: \ If in a given step of the training process the value of the objective function (in this case, the mean squared error) is worse (larger) than the value that was returned by the same step of the previous trials, then the current trial is not worthy to complete and the training proccess is terminated. This way, unpromising trials are early detected and pruned, saving computational resources and time. More accurately, the condition to prune a trial is that the value of the objective function in a given step must be larger than the median of the values of the same step in the previous trials. \

In order to divide the training process in several steps, we need to implement the partial_fit function. This function does not train the model with the whole dataset at once, but adding incremental chunks of data instead. This is how we can detect unpromising trials and stop the training before all the chunks are added. However, the partial_fit function can just be implemented to some algorithms that can learn incrementally. This is the reason we have selected stochastic gradient descent regressor for this exercise. Hence, the hyper-parameter we need to optimize in this case is alpha. As our aim is to just to show how the pruning algorithm works, we will use reduced training and testing sets.\

We see that the code is very similar to the one in the above-mentioned section. Nevertheless, we find the following changes: \ 1) Inside the objective function we perform a loop of 20 iterations corresponding to the number of steps the training process is going to take. \ 2) Inside this loop we evaluate the model resulted from each step and we save the value of the error (score) of each step (trial.report). \ 3) The pruner is called at the end of each step to inspect whether the trial should be pruned or not. \ 4) The pruner is specified in the study session. \

In the output the pruned trials appear explicitly. \

Graphycal analysis with optuna

Optuna offers a variety of graphical tools that help us visualize the optimization process. In this section we will discuss and implement some of them. For this purpose, the study session from section 3.1.4 will be used. Also, all the below-mentioned functions can use matplotlib as a backend.

In first place, we will use optuna.visualization.plot_optimization_history. Its output is a plot of the value of the objective function for each trial (blue dots). It also shows the current best value as a red line. This red line changes its position each time a new trial finds a better combination of hyperparameters. \

Optuna also allows to study the importance of each parameter for the estimation of the objective function. Such importances can be visualized executing optuna.visualization.plot_param_importances.

From this plot we can conclude that the maximum depth is the least important hyper-parameter, while the learning rate is the most important one. This means that spending computational resources on searching the optimal learning rate would be much more efficient and would give similar results than searching for the whole optimal hyper-parameter space. \

Now we wil implement optuna.visualization.plot_slice. This function plots the value of the objective function againts the value of each hyper-parameter in each trial. This function allows to introduce a list of parameters to visualize, but by default all parameters are shown.

By using optuna.visualization.matplotlib.plot_parallel_coordinate it is possible to get a parallel coodinates plot (PCP). PCP draws coordinates in parallel axes and connects them with straight lines. The variables (hyper-parameters and objective function) are drawn into the horizontal axis and its values are mapped onto the vertical axis. Each line represents a trial.\ This graph helps to visualize if the election of any hyper-parameters is somehow skewed .

Finally, we will make use of optuna.visualization.plot_contour. This command deploys parameter relationship as contour plot. The lighter areas show the values of the pair of hyper-parameter that minimizes the objective function.

Optuna allows to plot intermediate values of all trials in a study. Each step of the training process is drawn in the horizontal axis and the corresponding value of the objective function is mapped in the vertical axis. Every line represents a trial. \ To visualize it we will use the study session from section 3.1.5.

3.1.7 Models comparison

The results from hyper-parameter tuning are summed up in the following table:

Method Best hyper-parameters New RMSE Old RMSE New exec. time (s) Old exec.time (s)
KNN Regressor Neighbours: 8 0.1919 0.1968 0.8218 0.0480
Decision Tree Regressor Max depth: 11, Min samples split: 17 0.1996 0.2282 0.5258 0.0558
Extra tree Regressor Max depth: 12, Min samples split: 2 0.1461 0.1545 36.5019 2.5495
HistGradient Regressor Max depth: 11, Learning rate: 0.04, Max iter: 532 0.1449 0.1528 81.4344 2.1319

As expected, the performance of all models has been improved. Also, we can see that the execution time increases with the complexity of the model. \ However, the use of the optimal hyper-parameter in KNN just improves the performance a 0.4% with respect to the previous model. Also, the time spent on the search plus model training takes 18 times more than the computation of the model trained with default hyper-parameters. So, the final result is a model that does not work much better and at cost of a longer execution time. Tuning hyper-parameters in this case is not beneficial. \ Hyper-parameter tuning in decision tree regressor achieves improvement with respect to the old model, but the algorithm takes 9 times longer. Also, the performance of this new model is still the worst among the studied models. \ On the other hand, Bayes search (implemented by both ways), gets a good improvement in the models given by extra tree regresor and histgradient regressor, respectively. The time taken in both cases is much longer than in the previous models, but now, by contrast, the extra time results in a great improvement of the model. \ Based on these results, the most efficient model is the one given by extra tree regressor, as its performance is almost identical to the histgradient regressor, but with the advantage of being a simpler model that takes a shorter time when doing hyper-parameter tuning. However, we will compare both methods in sections 3.1.8 and 3.2.

3.1.8 CASH with extra trees and gradient boosting

As an exercise, we will use optuna in order to perform hyper-parameter tunning, including also the model as a parameter of the search space.

The model selected is histgradient regressor. This is not surprising, as we have seen before that the performance of this model is slightly better than the extra tree regressor. However, the selected hyper-parameters do not coincide with the ones we had previously found. Also, this new set results in a better performace.

3.2 Model evaluation

The histgradient boosting regressor and extra trees performed quite well, therefore, we will evaluate these two models in test sample in order to see their performance in test dataset. The reason to evaluate these two well-performed models in test sample is to find out whether these models also perform good in test sample or not.

After comparing two methods in 3.1.8 histgradient has been chosen , therefore, we will use the hyper-parameters of CASH for histgradient boosting regressor and hyper-parameters given by Bayes search for extra trees. Afterwards, we will compare the results and choose the best model.

To conclude, Hist Gradient Boosting model performed better than Extra Trees with very few difference. The test-rmse has been found 0.1256 for HGBM and 0.1453 for Extra Trees. This result makes clear that we should use Hist Gradient Boosting model to make the final model.

3.3 Final model

In this part, the sale price will be predicted for X_comp sample. Hist Gradient Boosting model will be used to predict.

We save the final results in .csv.

3.4 Extra - Halving Optimization Method

3.4.1 Halving Grid Search and Halving Random Search Explanation

Halving Grid Search Method

Halving Grid Search is an optimized version of Grid Search hyperparameter optimization. Halving Grid Search searches over a given list of hyperparameters using a sequential bisection approach. The search strategy begins evaluating all candidates on a small sample of data and iteratively selects the best candidates using increasingly larger numbers of samples. HalvingGridSearchCV, which is inside of the Scikit-Learn library, will be used to imply this method for KNN model.

Halving Random Search Method

Halving Randomized Search uses the same sequential halving approach and it is even more optimized than Grid Search. Unlike Halving Grid Search, it is not trained on all hyper-parameter combinations, but instead picks a random set of hyper-parameter combinations. HalvingRandomizedSeachCV, which is also inside of the Scikit-Learn library, will be used to imply this method for decision tree regressor model.

3.4.2 Implementing Halving Grid to KNN

Parameters definition:

Attributes definition:

Comparison between Grid Search CV and Halving Grid Search Methods

Both CV results are shown below:

3.4.3 Implementing Halving Random Search Method to Decision Tree Regressor Model

Comparison between Random Search CV and Halving Random Search Methods

Both CV results are shown below:

3.4.3 Using XGBoost

XGBoost is an optimized Gradient Boosting algorithm through parallel processing with threads, tree-pruning, handling missing values and regularization to avoid overfitting/bias.

Before using XGBModel, data is called again because in previous section, we have done the prediction and found y_comp. After calling the data again from the same excel file, all x_train, y_train, x_test and y_test transformed to sparse matrises.

Hyper Parameter Tuning

Hyper Parameter Tuning with Optuna

In Optuna, the best model has been found with 0.1362 RMSE score with following best hyper parameters. We will test the selected model in model evaluation section.

Hyper Parameter Tuning with Halving Random Search for XGBModel

In Halving RandomSearchCV, the best model has been found with 0.1376 RMSE which is larger than the RMSE achieved by Optuna. We will test both selected models in model evaluation section to see which one is best for the test sample.

Note: In halving RandomSearchSV, the hyper-parameters grid were not given randomly as in Optuna.

XGBModel Evaluation

XGBModel Evaluation with Optuna's best hyper-parameters

Even though the best score in Optuna with the training dataset has been found less than in HalvingRandomSearchCV, as seen in above results, the chosen model with less training RMSE could not work very good in test sample. This is beacuse the chosen parameters causes overfitting. Due to that reason, the selected model has higher RMSE in evaluation sample (0.1455).

XGBModel Evaluation with Halving Random Search Hyper-parameters

By using RandomSearchCV for hyper parameter tuning of XGBModel, we got higher RMSE score for training sample compared to Optuna. However, the test-RMSE has been found lower than Optuna's test-RMSE score.

Even though the train-RMSE score was higher than in the model using Optuna, the selected model from HalvingRandomSearchCV worked better in test sample. Test-rmse has been calculated 0.1233. Therefore, for prediction, the model with less eval-RMSE will be used.

Feature Importance with Weight and Gain Values

In XGBoost, it is possible to visualize each feature importance with its weight, which is useful to distinguish the relevant attributes.

Plotting Test and Train Samples

Final Model - Prediction

Conclusions

For improvement, XGBoost model has been used to predict the Sale Price. Best hyper-parameters for XGBoost model have been tried. These have been found by HalvingRandomSearchCV and Optuna. After doing model evaluation in test sample, XGBoost model has been executed with the best hyper-parameter given by Halving random seach CV. The test-RMSE score calculated is 0.1234 which is quite similar but smaller than the solution found with histgradient boosting model (0.1256).

References

Optuna documentation \ https://optuna.readthedocs.io/ \ https://scikit-learn.org/0.15/modules/scaling_strategies.html \ https://medium.com/optuna/an-introduction-to-the-implementation-of-optuna-a-hyperparameter-optimization-framework \ https://scikit-learn.org/stable/modules/grid_search.html#searching-for-optimal-parameters-with-successive-halving \ https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.