I am doing the Coursera Machine Learning course and there is something I am struggling with Gradient Descent applied to the linear regression.

So basically given the definition of the cost function (Mean Squared Error): $J(\theta) = \dfrac {1}{2m} \displaystyle \sum _{i=1}^m \left ( \hat{y}_{i}- y_{i} \right)^2 = \dfrac {1}{2m} \displaystyle \sum _{i=1}^m \left (h_\theta (x_{i}) - y_{i} \right)^2$

And the basic steps below (in a 1D case):

Repeat until convergence

  • $temp0 = \theta_0 - \alpha \frac{\partial}{\partial \theta_0} J(\theta_0, \theta_1)$
  • $temp1 = \theta_1 - \alpha \frac{\partial}{\partial \theta_1} J(\theta_0, \theta_1)$
  • $\theta_0 = temp0$
  • $\theta_1 = temp1$

At some point it is shown that:

  • $\theta_0 = \theta_0 - \alpha \frac{1}{m} \sum\limits_{i=1}^{m}(h_\theta(x_{i}) - y_{i})$
  • $\theta_1 = \theta_1 - \alpha \frac{1}{m} \sum\limits_{i=1}^{m}\left((h_\theta(x_{i}) - y_{i}) x_{i}\right)$

With a bit of explanation but based on $m$ = one single element:

  1. $\frac{\partial}{\partial \theta_j} J(\theta)=\frac{\partial}{\partial \theta_j} \frac{1}{2} (h_{\theta}(x) - y)^2$

  2. $\Leftrightarrow 2 \frac{1}{2} (h_{\theta}(x) - y) \frac{\partial}{\partial \theta_j} (h_{\theta}(x) - y) = (h_{\theta}(x) - y) \frac{\partial}{\partial \theta_j}(\sum_{i=0}^{n} \theta_i x_i - y )$

  3. $\Rightarrow (h_{\theta}(x) - y) x_j$

I don't really see how we end up on step 2 from step 1, any idea?

I feel there is something missing, even if we put: $h_\theta(x)=\theta_0 + \theta_1 x_i$. I mean the partial derivative is applied on $j$...

  • $\begingroup$ Whoops I had thought you asked step 3 from step 2, and answered incorrectly. $\endgroup$ – Davis Yoshida Feb 16 '18 at 22:25
  • $\begingroup$ @DavisYoshida yeah I am concerned about the first form of the partial derivative $\endgroup$ – Ehouarn Perret Feb 16 '18 at 22:27

This is the chain rule of differentiation being applied. If you have something like $f(g(\theta_j))$, its (ordinary) derivative with respect to $\theta_j$ is $f'(g(\theta_j)g'(\theta_j) $, where $f'$ and $g'$ are the derivatives of $f$ and $g$.

The same applies here, where $f(t) = \frac{1}{2}t^2$ and $g(\theta_j) = h_{\theta_1, \dots, \theta_j, \dots, \theta_m}(x)$ with all the other $\theta_{i\neq j}$ being considered as fixed. It's just a little bit weird since $h$ is not written as being a function of $\theta$.

We have $f'(t) = t$ and $g'(\theta_j) =\frac{\partial}{\partial \theta_j} (h_\theta(x) - y)$

Applying the chain rule:

$\frac{\partial}{\partial \theta_j} \frac{1}{2}(h_\theta(x) - y)^2 = f'(g(\theta_j))g'(\theta_j)\\ = g(\theta_j)g'(\theta_j)\\ = (h_\theta(x) - y)\frac{\partial}{\partial \theta_j} (h_\theta(x) - y)$

  • $\begingroup$ I also thought about that but I'm a bit annoyed about the j variable which is not necessarily in h_theta(x). Second thought \theta_j is actually in but can be as a constant whether when j = 0 or j = 1. $\endgroup$ – Ehouarn Perret Feb 17 '18 at 1:33
  • 1
    $\begingroup$ $j$ is just any constant index (i.e. $1, 2, \dots, n$), and $\theta_j$ is the element of the vector $\theta$ which we are computing the derivative for. $\endgroup$ – Davis Yoshida Feb 18 '18 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.