I am trying to fully understand stochastic gradient descent and I am having a hard time knowing if I fully grasp the concept. I know it is an algorithm used in data science and supervised/unsupervised learning. I also know that it maps the OLS to a radar map. I am wanting to use it one day on one of my regression analysis projects. I want to be able to incorporate the first step being the minimum of the standard errors? stochastic sum for minimization

Please let me know how this is used in regression analysis!

Thank you!


Gradient descent is a general method for finding the local minimum of a derivable function $E$.

This function is usually an error function, that is, a function that represents how wrong your model is at explaining the data it sees.

For example, suppose you wanted to estimate the height of a person in function of its weight. You may have some data collected of height and weight pairs, and suspect that the relation between both of them is linear, so $height_\theta(weight)=weight*\theta_1 + \theta_0$.

Your goal here would be to find the parameters $\theta$ that better explain the data, so you might define your error function as $\frac{1}{n}\sum E(\theta, Data_i)=\frac{1}{n}\sum (height_\theta(weight_i)- actualHeight_i)^2$.

The better your predictions using your model are, the lower the value of the error function $E$.

Now you may start with a random $\theta$.

In order to improve your estimate, compute the gradient of the error function $E$ with respect to $\theta$ for a given data point and nudge the value of theta in the opposite of that direction.

Since the gradient points toward where the function increases, by moving $\theta$ the other way you will reduce the error (as described by the error fucntion E) and thus improve your model.

Do this for all the data points you have and iterate until your model is good enough for your purposes.

That is stochastic gradient descent applied to supervised machine learning. As an aside, you could also compute the gradient of the mean of the errors for all the data points in your data sets with respect to $\theta$, instead of going one by one.

That is called batch gradient descent, and it's more used in machine learning due to its capability to exploit multiple cores, albeit it has the problem that it cannot process data "on the fly".

This is a pretty rough introduction to the concept, so you may want to check other material. I recommend Andrew Ng's course on machine learning in Coursera


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.