3
$\begingroup$

In Goodfellow et al.'s Deep Learning, on page 245, when proving the equivalence of early stopping and $L_2$ regularization,

The equation (7.41) is given as $$ \mathbf{Q}^T \tilde{\mathbf{w}} = (\Lambda + \alpha\mathbf{I})^{-1}\Lambda\mathbf{Q}^T\mathbf{w}^* $$ And it is rearranged (Equation (7.42)) to $$ \mathbf{Q}^T \tilde{\mathbf{w}} = [\mathbf{I} - (\Lambda + \alpha\mathbf{I})^{-1}\alpha] \mathbf{Q}^T\mathbf{w}^* $$ Can anyone please show me how this rearrangement is done? Thank you.

$\endgroup$

1 Answer 1

2
$\begingroup$

This is just an application of the Woodbury matrix identity. $$(\Lambda + \alpha \mathbf{I})^{-1} = \Lambda^{-1}-\Lambda^{-1}(\frac{1}{\alpha}\mathbf{I}+\Lambda^{-1})^{-1}\Lambda^{-1}.$$ Consequently, $$(\Lambda + \alpha \mathbf{I})^{-1}\Lambda = \Lambda^{-1}\Lambda-\Lambda^{-1}(\frac{1}{\alpha}\mathbf{I}+\Lambda^{-1})^{-1}\Lambda^{-1}\Lambda=\mathbf{1}-\Lambda^{-1}(\frac{1}{\alpha}\mathbf{I}+\Lambda^{-1})^{-1}.$$ Since $(AB)^{-1}=B^{-1}A^{-1}$, we can rewrite the last term: $$\Lambda^{-1}(\frac{1}{\alpha}\mathbf{I}+\Lambda^{-1})^{-1}=(\frac{1}{\alpha}\Lambda+\mathbf{1})^{-1}=(\frac{1}{\alpha}(\Lambda + \alpha\mathbf{1}))^{-1}=\alpha(\Lambda + \alpha\mathbf{1})^{-1}.$$

Putting it all together, we have $$(\Lambda + \alpha \mathbf{I})^{-1}\Lambda = \mathbf{1}-\alpha(\Lambda + \alpha\mathbf{1})^{-1}$$ and the rest follows easily.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .