1
$\begingroup$

I'm confused by multiple representations of the partial derivatives of Linear Regression cost function.

This is the MSE cost function of Linear Regression. Here $h_\theta(x) = \theta_0+\theta_1x$ .

\begin{aligned}J(\theta_0,\theta_1) &= \frac{1}{m}\displaystyle\sum_{i=1}^m(h_\theta(x^{(i)}) - y^{(i)})^2\\J(\theta_0,\theta_1) &= \frac{1}{m}\displaystyle\sum_{i=1}^m(\theta_0 + \theta_1x^{(i)} - y^{(i)})^2\end{aligned}

Are these the correct partial derivatives of above MSE cost function of Linear Regression with respect to $\theta_1, \theta_0$? If there's any mistake please correct me.

\begin{aligned}\frac{dJ}{d\theta_1} &= \frac{-2}{m}\displaystyle\sum_{i=1}^m(x^{(i)}).(\theta_0 + \theta_1x^{(i)} - y^{(i)})\\ \frac{dJ}{d\theta_0} &= \frac{-2}{m}\displaystyle\sum_{i=1}^m(\theta_0 + \theta_1x^{(i)} - y^{(i)})\end{aligned}

$\endgroup$
  • $\begingroup$ Yes, except the minus sign. But this is not important since you set them equal to $0$ $\endgroup$ – callculus Mar 17 at 23:40
  • $\begingroup$ @callculus So it's $\frac{2}{m}$ rather than $\frac{-2}{m}$ for both the cases. I couldn't get what you meant by "you set them equal to 0". $\endgroup$ – user214 Mar 17 at 23:45
  • $\begingroup$ Are you still interested in an answer? $\endgroup$ – callculus Mar 18 at 2:15
  • $\begingroup$ @callculus Yes. I'd appreciate it. $\endgroup$ – user214 Mar 18 at 2:29
  • 1
    $\begingroup$ usee214: Great, love it. $\endgroup$ – callculus Mar 18 at 15:38
3
$\begingroup$

The derivatives are almost correct, but instead of a minus sign, you should have a plus sign. The minus sign is there if we differentiate

$$J = \dfrac{1}{m}\sum_{i=1}^m\left[y_i-\theta_0-\theta_1 x_i\right]^2$$

If we calculate the partial derivatives we obtain

$$\dfrac{\partial J}{\partial \theta_0}=\frac{2}{m}\sum_{i=1}^{m}[y_i-\theta_0-\theta_1x_i]\cdot\left[-1 \right]$$ $$\dfrac{\partial J}{\partial \theta_1}=\frac{2}{m}\sum_{i=1}^{m}[y_i-\theta_0-\theta_1x_i]\cdot\left[-x_i \right]$$

In order to find the extremum of the cost function $J$ (we seek to minimize it) we need to set these partial derivatives equal to $0$ $$\dfrac{\partial J}{\partial \theta_0}=\frac{2}{m}\sum_{i=1}^{m}[y_i-\theta_0-\theta_1x_i]\cdot\left[-1 \right]=0$$ $$\implies \sum_{i=1}^{m}[y_i-\theta_0-\theta_1x_i]=0$$ $$\dfrac{\partial J}{\partial \theta_1}=\frac{2}{m}\sum_{i=1}^{m}[y_i-\theta_0-\theta_1x_i]\cdot\left[-x_i \right]=0$$ $$\implies \sum_{i=1}^{m}[y_i-\theta_0-\theta_1x_i]\cdot\left[x_i\right] = 0.$$

As we divide by $ -2/m $ for both cases we will obtain the same result. If you had $ +2/m $ then you would divide by $ 2/m $ and still obtain the same equations as stated above. If the equation that we need to solve are identical the solutions will also be identical.

$\endgroup$
  • $\begingroup$ Can you please include the corrected formula in your answer? $\endgroup$ – user214 Mar 17 at 23:53
  • $\begingroup$ You just have to multipy your partial derivatives by $(-1)$. Both ways lead to the same result. $\endgroup$ – MachineLearner Mar 18 at 6:22
  • $\begingroup$ I'm trying to build a Stochastic Gradient Descent. So can I use 2/m insted of -2/m and calculate the gradients right? $\endgroup$ – user214 Mar 18 at 10:00
  • 1
    $\begingroup$ @user214: In the end, the plus or minus does not make a difference, because you set the derivatives equal to zero. But your code could irritate other people. That is why you should use $2/m$ instead of the wrong $-2/m$ (but which leads to the same correct result) as a factor. $\endgroup$ – MachineLearner Mar 18 at 10:04
  • 1
    $\begingroup$ @user214: I added more details. You can try it on your own for the correct version and for the wrong version. You will see that we obtain the same result if you solve for $\theta_0$ and $\theta_1$. $\endgroup$ – MachineLearner Mar 18 at 11:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.