1
$\begingroup$

I post this question here because I think this is a calculus problem

I'm a software engineer, and I have just started a Udacity's nanodegree of deep learning.

I have also worked my way through Stanford professor Andrew Ng's online course on machine learning and now I'm comparing.

I have a big doubt about Gradient Descent with sigmoid function because on Andrew Ng's course it is different from the one I see on Udacity's nanodegree.

From Andrew Ng's course, gradient descent is (First formula): enter image description here

But, from Udacity's nanodegree is (Second formula):

enter image description here

Note: first picture is from this video, and second picture is for this other video.

But in this CS229 course notes from Andrew Ng's, on page 18, I have found the demonstration from Andrew Ng's gradient ascent formula. I only add it here as a demonstration because I haven't found this derivative process for gradient descent and I don't know how to do it:
enter image description here

Note: the formula above is for Gradient Ascent.

I'm not sure if I have understood everything, but in this derivative I see have the derivative from f function disappears (f function is the sigmoid function).

But in Udacity's nanodegree they continue using the sigmoid's derivative in their gradient descent.

The difference between first formula and second formula is the derivative term.

Are the two formula equivalents?
Where can I find all partial derivative steps for that partial derivative?

$\endgroup$
  • $\begingroup$ The first picture maximizes the log-likelihood of the data and the second picture minimizes the squared loss function. In short the difference is because of different objectives. $\endgroup$ – user94300 Feb 14 '17 at 16:34
-2
$\begingroup$

Please take a look at this part of Machine learning course on Coursera which can help you with your question: https://www.coursera.org/learn/machine-learning/lecture/MtEaZ/simplified-cost-function-and-gradient-descent In this part, the lecturer is showing the result of derivative in gradient descent for logistic regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.