I am struggeling to derive the squared-bias and variance based on an eigendecomposition for the OLS-procedure.

The model

Consider the univariate model $y_i = f(x_i) + \epsilon_i, \ i = 1, \dots, n$ with $E(\epsilon_i)=0$ and $Var(\epsilon_i)=\sigma^2$.

Let $S$ be a projection matrix (obtained by OLS) such that the fitted values are $\hat{Y}=SY$.

Let $S=ODO^T$ with $OO^T=I$ and $D=diag(d_1, \dots, d_n)$ be an eigendecomposition of $S$.

I derived a formula for the squared bias and variance:

(1) $Var(\cdot{}) = \sigma^2n^{-1} \sum_{k=1}^{n}d_k^2$

(2) $bias^2 = n^{-1}f^TO diag((1-d_k)^2)O^Tf,$ where $f = (f(x_1), \dots, f(x_n))^T$


Since the eigenvalues are $d_1=d_2=1$ (univariate model with slope and intercept) and $d_k=0 \ \forall k =3, \dots, n$, I obtain from (1) the variance of the OLS procedure, i.e. $Var(\cdot{}) \approx \sigma^2 2/n$ (which seems to be correct).

However, I am not sure what the squared bias is (based on the above formula). I thought it is 0 but cannot see it as the entries in the diagonal matrix from (2) $diag((1-d_k)^2)$ for $k=3, \dots, n$ are not zero but one. I know that the first two eigenvectors (columns) of the matrix $O$ form a linear subspace. What about the remaining $(n-2)$-eigenvectors?

Sketch of derivation of above expressions

For (1), simplify $Cov(SY)$ and then take the trace to obtain the variance

For(2), $(E(SY)-f)^T(E(SY)-f)$ and simplify using orthonormality of $O$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.