Question

Given a figure visualising the decision boundaries learned by a SVM, explain which type of kernel function is used. Describe the hyper-parameter(s) of that kernel function and explain the meaning of those hyper-parameter(s).

OD 0990 Training dataset Decision boundaries learned by SVM

Answer 1

The Support Vector Machine is a supervised learning algorithm mostly used for classification purposes but can also be used for regression purposes. The main idea is that the algorithm tries to find the optimal hyperplane that can be used to classify new data points, based on the labeled data (training data) the hyperplane is a simple line, in two dimensions.

Usually the learning algorithm tries to learn the most common characteristics (which distinguishes one class from another) of the class and the classification is based on the representative characteristics learned (so classification is based on differences between classes). The SVM is working the other way around. The most similar examples are found between classes. Those are the support vectors.

It is important to select the right kernel, because if the transformation is wrong, then the model can have very bad results. As a thumb rule, always check if you have linear data, and use linear SVM (linear kernel) in that case. Linear SVM is a parametric model but it's not an RBF kernel SVM, so the latter's complexity grows with the size of the training set. Not only is it more expensive to train an RBF kernel SVM, you also have to keep the kernel matrix around, and the projection into this higher-dimensional "infinite" space where the data becomes linearly separable is also more expensive during prediction. You also have more hyperparameters to tune, so the selection of models is more expensive too. And lastly, it's easier to overfit a complex model.

A hyper parameter in machine learning is a parameter which is used to control the learning process by its value. By contrast, other parameter values (usually node weights) are derived via training. Hyper parameters can be classified as model hyperparameters that can not be deduced when fitting the machine to the training set because they refer to the selection task of the model or the hyper parameters algorithm That in principle has no influence on model performance but has an impact on the speed and quality of the learning process. The topology and size of a neural network is an example of a hyper parameter model. Examples of hyper parameters for algorithms are learning rate and mini-batch size.