Let $A$ be some matrix over $\mathbb R$.

Is the function $f(\lambda)=\lVert A(A^TA+\lambda I)^{-1}A^Ty \rVert_2$ decreasing for $\lambda > 0$? Here $y$ is an arbitrary real vector of the correct size.

This function has some statistical significance (the solution of ridge regression), where it seems plausible that "more regularization gives smaller coefficients", i.e. that even each component separately before taking the norm is decreasing in absolute value. In my case $A$ is a binary matrix with some additional structure, but I presume it does not matter.

It is true in the 1-dimensional case, and when $A^TA$ is diagonal (again each component separately is decreasing). How do I approach the general case?

  • $\begingroup$ The solution of ridge regression is $(A^TA + \lambda I)^{-1} A^Ty$ whereas here there is an extra $A$ at the front. Is this normal ? $\endgroup$
    – Digitallis
    May 15 at 9:48
  • $\begingroup$ Yes, I want the values and not the coefficients, but if you prove yours then mine follows. $\endgroup$
    – Pachirisu
    May 15 at 10:08
  • $\begingroup$ Have you tried using the fact that $A^TA$ is real symmetric ---> it is diagonalisable. This seems obvious to try, especially if you say the result is easy when $A^TA$ is diagonal $\endgroup$
    – Digitallis
    May 15 at 10:17
  • $\begingroup$ It is not true for every $\lambda$, not even in the 1d case. Not sure how you arrived at this conclusion. $\endgroup$
    – V.S.e.H.
    May 15 at 11:08
  • 2
    $\begingroup$ This can be reduced to the diagonal case by using the SVD of $A$. $\endgroup$
    – Hyperplane
    May 15 at 11:26


You must log in to answer this question.

Browse other questions tagged .