How can I derive A in this matrix represented equation(a closed form solution for ridge regression) $$\ a= (X^T X + αI)^ {−1} X^T y $$ (-1 here is inverse) and $a = X^TA$ I have to get $$\ α^{-1}(y-Xa) .$$ But I am not sure how can i get to it . TIA!
$\begingroup$
$\endgroup$
12
-
$\begingroup$ Is that supposed to be an inverse symbol, a {-1} exponent? $\endgroup$– RobertTheTutorApr 11, 2021 at 18:45
-
$\begingroup$ yes, edited, thanks- it is inverse $\endgroup$– jack vorobeyApr 11, 2021 at 18:45
-
$\begingroup$ Is "a" a matrix? It looks as if that is how you are using it; I'm just used to seeing uppercase letters for matrices and lowercase for scalars. $\endgroup$– RobertTheTutorApr 11, 2021 at 18:47
-
$\begingroup$ it is vector here $\endgroup$– jack vorobeyApr 11, 2021 at 18:48
-
1$\begingroup$ If $X$ is $n \times 1$, a column vector, then $X^T$ is $1 \times n$, a row vector, and $X^TX$ is a scalar. And $\alpha$ you say is a scalar. So doesn't that make $I$ a scalar as well? Please edit your question itself to show what are what sizes. $\endgroup$– RobertTheTutorApr 11, 2021 at 19:07
|
Show 7 more comments
1 Answer
$\begingroup$
$\endgroup$
1
Note that $\ X^TX+αI$ and its inverse are scalars. This is because $X$ is an $n \times 1$ matrix, so $X^T$ is $1 \times n$ and thus $X^TX$ is $1 \times 1$. Thus, we can move this around without worring about its order, unlike matrices.
Use the hint as the first step (after substituting the expression for $a$): \begin{align} \ a&= (X^T X + \alpha I)^ {−1} X^T y \\ X^TA &= (X^T X + \alpha I)^ {−1} X^T y \\ (X^T X + \alpha I)X^TA &= X^T y \\ (\alpha^{-1}X^T X + I)X^TA &= \alpha^{-1}X^T y \\ \alpha^{-1}X^T X X^T A + X^T A&= \alpha^{-1}X^T y \\ X^T A&= \alpha^{-1}X^T y - \alpha^{-1}X^T X X^T A \\ (X^T)^{-1}X^T A&= \alpha^{-1}(X^T)^{-1}X^T y - \alpha^{-1}(X^T)^{-1}X^T X X^T A \\ A&= \alpha^{-1} y - \alpha^{-1} X X^T A \\ A&= \alpha^{-1} (y - X X^T A) \\ A&= \alpha^{-1} (y - X a). \end{align} Of course, this is only valid if
- $X^T X + \alpha I \neq 0$, and
- $X^T$ is invertible, which happens if and only if $X$ is invertible i.e. non-singular.
-
$\begingroup$ You said X is a vector. How can it be invertible? $\endgroup$ Apr 11, 2021 at 20:02