Question

Machine Learning

4·Let (X.J) be a instance-label set with χ e Rn and let χ1ram be a sample set (of training data). I (is linearly separable, then there is some logistic regression model Hsig that will produce a loss of 0 on xtrain (a) Explain why the last sentence above is true. (b) If training a logistic regression model on χtran produces a loss of 0, explain why it is not necessarily the case that (x,is linearly separable. In the same situation, where the loss is 0 on trai, even if it is the case that (x, is linearly separable, will the logistic regression model that was trained on xrain necessarily predict the correct label for every z E X? Explain why or why not.

Answer 1

(X, Y) is an instance labelled set. If (x,y) is linearly separable then it means we can have a line which will separate this set into two groups having different properties. No element of one group will be on another side of the line.

So we can have a logistic regression model Hsig which will produce zero loss, loss being error calculated if one element is on another side of the line. So any set X_train from the full set will also follow the same property and it will also not have any loss.

b)

If the logistic regression model produces zero loss on the training set, we cannot say anything about the set being linearly separable or not, because we are not guaranteed that the test set will also produce zero loss.

Similarly, even if the loss is zero on X_train and we have linearly separable set we are not sure that the regression line is the same line which divides the set. It can produce some loss while testing and we can eliminate this loss by taking more iterations of the training.