Motivation for gradient descent method over OLS/MLE for simple linear regression? I am beginner in machine learning and I am currently trying to find the motivation for gradient descent method. I am confused why we want to employ gradient descent method for linear regression? I see the cost function the same as the OLS function, and gradient descent method here actually takes more effort than simply getting the derivatives equal one. Then why we always try to use gradient descent here? I am when the model gets more complicated , and also when we make more assumptions on the prior distribution of the theta(parameters). The optimization problem will become much more complicated. Then will gradient descent method still survive in terms of this? And OLS/MLE method will not be able to predict the parameters? I see OLS as minimize the cost, and the MLE method as maximize the prob, which is in essence the same.(reference http://www.cs.ubc.ca/~nando/540-2013/lectures/l3.pdf) Should I think gradient descent method as a improvement from the OLS method, while the E-M method(maximize the expected likelihood) as a imporvement from the MLE method. Thanks in advance!
 A: In the case of linear regression, we want to obtain estimates of the coefficients $\theta_1, \theta_2 $ where:
$$y = \theta_0 + \theta_1x_1 + ...+ \theta_nx_n + \epsilon$$
where $\epsilon $ is the error modeled as $N(0,\sigma^2)$.  
The optimal parameters will then be
$\hat\theta = argmin_\theta L(\theta)$
where L is the OLS function you want to minimize
$L(\theta) = \frac{1}{m}\sum_{i=1}^{m}(y_i - \theta^Tx_i)^2$
On finding the first derivative of this function, equating it to zero, you will find the optimal vector of $\theta$ as:
$\hat\theta = (XX^T)^{-1}XY$
Now the problem here is that matrix inversion is a very expensive operation and so for problems with a lot of data when X and Y are large matrices, it is better to use the gradient descent method even though it is a heuristic that gets us the locally optimal parameter values.
I didn't quite understand this part of your question:
"I am when the model gets more complicated , and also when we make more assumptions on the prior distribution of the theta(parameters). The optimization problem will become much more complicated."
If you re-phrase it, I will edit my response and try to answer that part as well.
