Question

After seeing the problem, your classmate Alice immediately argues that we can apply a linear regression model, as the labels are numbers from 0-9, very similar to the example we learned from Unit 1. Though being a little doubtful, you decide to have a try and start simple by using the raw pixel values of each image as features. Alice wrote a skeleton code run_linear_regression_on_MNIST in main.py, but she needs your help to complete the code and make the model work. Closed Form Solution of Linear Regression 5.0/5.0 points (graded) To solve the linear regression problem, you recall the linear regression has a closed form solution: 0 = (x+ x + 11)-XTY where I is the identity matrix. Write a function closed_form that computes this closed form solution given the features X, labels Y and the regularization parameter 2.
I only need help for the last question "What Went Wrong?". Thank you!

Answer 1

It should be b) the loss function related to closed form solution is inadequate for this problem.

the loss function used in regression is least square which does not work well for classification problems. instead you should use logistic loss function for this task.

here is a simple example for binary classfication which serves same explanation for multiclass classification like above

Suppose you have a binary classification problem, where you want to label data points as +1 or -1. The way you typically set up a classification problem is as follows — for a given data point x, you calculate a function f(x) and if that function value is large enough, you classify that point as +1, otherwise as -1. Now, if you use one of the standard classification losses like hinge loss or logistic loss, then for large values of f(x), if your true label is +1, then these losses go to zero. That is, you do not want to penalize correctly labeled points, and therefore keep them away from interfering with your parameter updates. However, with mean squared error, if your f(x) is anything other than +1, you incur a loss, and even correctly classified points influence the gradients, making learning harder.