Proof that loss function for linear regression is an ellipsoid I cannot seem to find a proof that $f(\mathbf{\beta}) = \left\lVert \mathbf{y}-\mathbf{X} \mathbf{\beta} \right\rVert^2$ is an ellipsoid, centered at the OLS solution $\hat{\beta}$. Can anyone show how to convert it to the quadratic form of a general ellipsoid, i.e. $(\beta - \hat{\beta})^T \mathbf{A} (\beta - \hat{\beta})$, where $\mathbf{A}$ is positive definite?
 A: The function $(\beta - \hat{\beta})^\top A (\beta - \hat{\beta})$ is zero when $\beta = \hat{\beta}$, yet $f(\hat{\beta}) = \|y - X \hat{\beta}\|^2$ might not be zero.
The form you are looking for is $(\beta - \hat{\beta})^\top A (\beta - \hat{\beta}) + c$ for some vector $c$ (which will turn out to be $\|y - X \hat{\beta}\|^2$). This shows the level sets of $f$ are ellipsoids centered at $\hat{\beta}$.

$\hat{\beta}$ is characterized by $$X^\top y = X^\top X \hat{\beta}.\tag{$*$}$$
Using this we have
$$
\begin{align}
(\beta - \hat{\beta})^\top X^\top X (\beta - \hat{\beta})
&= \beta^\top X^\top X \beta - 2 {\beta}^\top X^\top X \hat{\beta} + \hat{\beta}^\top X^\top X \hat{\beta}
\\
&= \beta^\top X^\top X \beta - 2 {\beta}^\top X^\top y + \hat{\beta}^\top X^\top y
\\
&= \beta^\top X^\top X \beta - 2 {\beta}^\top X^\top y+  y^\top y - y^\top(y - X\hat{\beta})
\\
&= \|y - X \beta\|^2 - y^\top (y - X\hat{\beta}).
\end{align}$$
So, $f(\beta) = (\beta - \hat{\beta})X^\top X (\beta - \hat{\beta}) + y^\top (y - X\hat{\beta})$.
Note that the additive term $y^\top (y - X \hat{\beta})$ is constant (since it does not depend on $\beta$), and can also be written as $\|y - X \hat{\beta}\|^2$ using the normal equation ($*$).
$$f(\beta) = (\beta - \hat{\beta})X^\top X (\beta - \hat{\beta}) + \|y - X \hat{\beta}\|^2.$$
