Question

- For the paramters ß of the linear model, your colleague proposes the new estimator: Bridge = (X'X + I)-?X'y. a. we derived the expected valu {'X--X'y for the linear model y ~ N(XB, oʻI). For the linear model, derive the expected value of Bridge. Simplfiy your answer as much as possible. b. In 3-5 sentences explain the implications of your result from part a.

Answer 1

Ans

(a)

Consider an experiment in which p characteristics of n samples are measured. The data from this experiment are denoted X, with X as above. The matrix X is called the design matrix. Additional information of the samples is available in the form of Y (also as above). The variable Y is generally referred to as the response variable. The aim of regression analysis is to explain Y in terms of X through a functional relationship like Yi = f(Xi,∗). When no prior knowledge on the form of f(·) is available, it is common to assume a linear relationship between X and Y. This assumption gives rise to the linear regression model:

                Yi = X_i,∗ β + ε_i                                        (1)

= β₁ X_i,1 + . . . + β_p X_i,p + ε_i .

In model (1) β = (β₁, . . . , β_p) ⊤ is the regression parameter. The parameter β_j , j = 1, . . . , p, represents the effect size of covariate j on the response. That is, for each unit change in covariate j (while keeping the other covariates fixed) the observed change in the response is equal to βj . The second summand on the right-hand side of the model, ε_i , is referred to as the error. It represents the part of the response not explained by the functional part X_i,∗β of the model (1). In contrast to the functional part, which is considered to be systematic (i.e. non-random), the error is assumed to be random. Consequently,

Y_i1,∗ need not equal Y_i2,∗for i₁ ≠ i₂, even if X_i1,∗ = X_i2,∗. To complete the formulation of model (1) we need to specify the probability distribution of ε_i. It is assumed that ε_i ∼ N (0, σ² ) and the ε_i are independent, i.e.:

02 Cov(Eis, Eix) = if if i1 = 12, ii + 12.

The randomness of ε_i implies that Y_i is also a random variable. In particular, Y_i is normally distributed, because ε_i ∼ N (0, σ² ) and Xi,∗ β is a non-random scalar. To specify the parameters of the distribution of Y_i we need to calculate its first two moments. Its expectation equals:

E(Y_i) = E(Xi,∗ β) + E(ε_i) = Xi,∗ β,

while its variance is:

Var(Y) = E{[Y; - E(Y:)]}} = E(Y) – [E(Y:)] = E(X+3)2 + 2€;X+B+)-(X3)2 = E(X) = Var(i) = 02.

Hence, Y_i ∼ N (Xi,∗ β, σ2 ). This formulation (in terms of the normal distribution) is equivalent to the formulation of model (1), as both capture the assumptions involved: the linearity of the functional part and the normality of the error.

Model (1) is often written in a more condensed matrix form:

Y = X β + ε,                                          (2)

where ε = (ε₁, ε₂, . . . , ε_n) ⊤ and distributed as ε ∼ N (0_p, σ2 I). As above model (2) can be expressed as a multivariate normal distribution: Y ∼ N (X β, σ² I).

Model (2) is a so-called hierarchical model.

Ans.

(b)

This terminology emphasizes that X and Y are not on a par, they play different roles in the model. The former is used to explain the latter. In model (1) X is referred as the explanatory or independent variable, while the variable Y is generally referred to as the response or dependent variable.

The covariates, the columns of X, may themselves be random. To apply the linear model they are temporarily assumed fixed. The linear regression model is then to be interpreted as Y | X ∼ N (X β, σ2 I).