Question

5. For ridge regression, we choose parameter estimators b which minimise i-1 j-0 where λ is a constant penalty parameter. (a) Show that these estimators are given by (b) Calculate the ridge regression estimates for the data from Q2 with penalty parameter 0.5. In order to avoid penalising some parameters unfairly, we must first scale every predictor variable so that it is standardised (mean 0, variance 1), and centre the response variable (mean 0), in which case an intercept parameter is not used. (Hint: This can be done with the scale function)

Answer 1

Let's build on what we know, which is that whenever the n × p model matrix is X, the response n- vector is y, and the parameter p-vector is β, the objective function (which is the sum of squares of residuals) is minimized when B solves the Normal equations Ridge regression adds another term to the objective function (usually after standardizing all variables in order to put them on a common footing), asking to minimize for some non-negative constant λ It is the sum of squares of the residuals plus a multiple of the sum of squares of the coefficients themselves (making it obvious that it has a global minimum). Because λ > 0, it has a positive square root v2-λ Consider the matrix X augmented with rows corresponding to v times the p x p identity matrix I X. When the vector y is similarly extended with p zeros at the end to y., the matrix product in the objective function adds p additional terms of the form (0 B-A to the original objective. Therefore From the form of the left hand expression it is immediate that the Normal equations are Because we adjoined zeros to the end of y, the right hand side is the same as X'y. On the left hand side v21 = λ| is added to the original X,X. Therefore the new Normal equations simplify to

Please post part b) again as Q2 data is not given