How can I derive A in this matrix represented equation(a closed form solution for ridge regression) $$\ a= (X^T X + αI)^ {−1} X^T y $$ (-1 here is inverse) and $a = X^TA$ I have to get $$\ α^{-1}(y-Xa) .$$ But I am not sure how can i get to it . TIA!

  • $\begingroup$ Is that supposed to be an inverse symbol, a {-1} exponent? $\endgroup$ Apr 11, 2021 at 18:45
  • $\begingroup$ yes, edited, thanks- it is inverse $\endgroup$ Apr 11, 2021 at 18:45
  • $\begingroup$ Is "a" a matrix? It looks as if that is how you are using it; I'm just used to seeing uppercase letters for matrices and lowercase for scalars. $\endgroup$ Apr 11, 2021 at 18:47
  • $\begingroup$ it is vector here $\endgroup$ Apr 11, 2021 at 18:48
  • 1
    $\begingroup$ If $X$ is $n \times 1$, a column vector, then $X^T$ is $1 \times n$, a row vector, and $X^TX$ is a scalar. And $\alpha$ you say is a scalar. So doesn't that make $I$ a scalar as well? Please edit your question itself to show what are what sizes. $\endgroup$ Apr 11, 2021 at 19:07

1 Answer 1


Note that $\ X^TX+αI$ and its inverse are scalars. This is because $X$ is an $n \times 1$ matrix, so $X^T$ is $1 \times n$ and thus $X^TX$ is $1 \times 1$. Thus, we can move this around without worring about its order, unlike matrices.

Use the hint as the first step (after substituting the expression for $a$): \begin{align} \ a&= (X^T X + \alpha I)^ {−1} X^T y \\ X^TA &= (X^T X + \alpha I)^ {−1} X^T y \\ (X^T X + \alpha I)X^TA &= X^T y \\ (\alpha^{-1}X^T X + I)X^TA &= \alpha^{-1}X^T y \\ \alpha^{-1}X^T X X^T A + X^T A&= \alpha^{-1}X^T y \\ X^T A&= \alpha^{-1}X^T y - \alpha^{-1}X^T X X^T A \\ (X^T)^{-1}X^T A&= \alpha^{-1}(X^T)^{-1}X^T y - \alpha^{-1}(X^T)^{-1}X^T X X^T A \\ A&= \alpha^{-1} y - \alpha^{-1} X X^T A \\ A&= \alpha^{-1} (y - X X^T A) \\ A&= \alpha^{-1} (y - X a). \end{align} Of course, this is only valid if

  • $X^T X + \alpha I \neq 0$, and
  • $X^T$ is invertible, which happens if and only if $X$ is invertible i.e. non-singular.
  • $\begingroup$ You said X is a vector. How can it be invertible? $\endgroup$ Apr 11, 2021 at 20:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .