0
$\begingroup$

What is the partial of the Ridge Regression Function? Does it provide insight to a gradient descent with ridge Regression? I am using Hands-on Machine Learning with Scikit Learn, Keras, and TensorFlow Geron.

$$\mathbf{Ridge \ Regression\ Equation} => J(\hat\theta) = MSE(\hat\theta) + \alpha\sum_{i=1}^n\theta_i^2 $$ $$\mathbf{ Gradient\ Descent\ Equation} => \frac{\delta}{\delta\theta_j}MSE=\frac{2}m\sum_{i=1}^m(\hat\theta^Tx^{(i)}-y^{(i)})x_j^{(i)}$$

$\endgroup$
0

1 Answer 1

1
$\begingroup$

Your ridge term is: $R = \alpha\sum_{i=1}^n\theta_i^2$

Its partial derivative can be computed using the power rule and the linearity of differentiation:

$$ \frac{\delta}{\delta \theta_j}R = 2\alpha\theta_j $$

You also asked for some insight, so here it is: In the context of gradient descent, this means that there's a force pushing each weight $\theta_j$ to get smaller. If a given weight doesn't have much effect on the loss function, then it will tend to get pushed towards a very small value. This can help to prevent overfitting, where weights with weak effects on the loss are pushed to unrealistically large values, just because it decreases the loss a little bit.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .