The mean squared error of Linear least squares solution Suppose I have a linear system $y=Ax+\varepsilon$ where $y\in\mathbb R^n$, $x\in\mathbb R^m$ and $A\in\mathbb R^{n\times m}$. I have known that the least squares solution for $x$ is $\hat{x}=A^+y$ where $A^+=(A^HA)^{-1}A^H$ is the pseudoinverse of $R$. However, I have read a book says that the mean square error (MSE) of the solution is (without proof)
$\|\hat{x}-x\|_2^2=\sigma^2\sum_k\frac{1}{\lambda_k}$
where $\lambda_k$ is the $k$-th eigenvalue of the matrix $A^HA$.
Could someone help me how to prove the equation. Thanks a lot!
 A: Assuming that $\varepsilon$ is distributed as $\mathcal{N}(0,\sigma^2 I)$, we have
\begin{align}
\text{MSE}=\mathbb{E}\|x-\hat{x}\|_2^2 &=\mathbb{E}\|(A^H A)^{-1}A^H y-\hat{x}\|_2^2\\
&=\mathbb{E}\|(A^{H}A)^{-1}A^H\varepsilon\|_2^2\\
&=\|(A^H A)^{-1}A^H\|_F^2\sigma^2,
\end{align}
where the second step uses $y=Ax+\varepsilon$ and the last step can be seen by straightforward calculation. Now, using $\|A\|_F^2=\operatorname{Tr}(A^H A)$, we get
\begin{align}
\text{MSE}&=\operatorname{Tr}(A^H A)^{-1}\sigma^2\\
&=\sum_i\frac{1}{\lambda_i}\sigma^2
\end{align}
where $\lambda_i$ is the $i^\text{th}$ eigenvalue of $A^H A$.
A: In comments you wrote: "could you elaborate more on the last step that transforming the 2-norm to the F-norm?"
I was going to reply to that in a comment, but what I wrote is more than twice as long as the 600-character limit.
$$ \begin{align} & \operatorname E \|(A^{H}A)^{-1}A^H\varepsilon\|_2^2 \\ {} \\ = {} & \operatorname E\Big( \big( (A^H A)^{-1} A^H\varepsilon \big)^H \big( (A^H A)^{-1} A^H \varepsilon \big) \Big) \\ {} \\ = {} & \operatorname E\Big( \big( \varepsilon^H A (A^H A)^{-1} \big) \big((A^HA)^{-1} A^H \varepsilon\big) \Big) \\ {} \\ = {} & \operatorname E \left( \operatorname{tr}\Big( \big( \varepsilon^H A (A^H A)^{-1} \big) \big((A^HA)^{-1} A^H \varepsilon\big) \Big) \right) \\
& \text{because the trace of a $1\times1$ matrix is its one entry} \\ {} \\
= {} & \operatorname E\left( \operatorname{tr} \Big( \varepsilon\varepsilon^H A(A^H A)^{-1} (A^H A)^{-1} A^H \Big) \right) \\
& \text{because } \operatorname{tr}(BC) = \operatorname{tr}(CB) \\ {} \\
= {} & \operatorname{tr} \left( \operatorname E \Big( \varepsilon\varepsilon^H A(A^H A)^{-1} (A^H A)^{-1} A^H \Big) \right) \\
& \text{because tr is linear} \\ {} \\
= {} & \operatorname{tr} \left( \Big( \operatorname E\big( \varepsilon \varepsilon^H \big)  \right) A(A^H A)^{-1} (A^H A)^{-1} A^H \Big) \\
& \text{because $\varepsilon$ is the only part that is random} \\ {} \\
= {} & \sigma^2 \operatorname{tr} \big( A(A^H A)^{-1} (A^H A)^{-1} A^H \big) \\
& \text{because the expected value is } \sigma^2 I_n \\ {} \\
= {} & \sigma^2 \operatorname{tr} \big( A^HA(A^H A)^{-1} (A^H A)^{-1} \big) \\
& \text{because } \operatorname{tr}(BC) = \operatorname{tr}(CB) \\ {} \\
= {} & \sigma^2 \operatorname{tr}( (A^H A)^{-1}) \end{align}$$
