Newton's Method vs Gradient Descent? Currently I am studying logistic regression.
I read online that Gradient Descent is a 1st-order optimisation algorithm, and that Newton's Method is a 2nd order optimisation algorithm.
Does that mean that Gradient Descent cannot be used for multivariate optimisation and that Newton's Method cannot be used for univariate optimisation? Or can Newton's Method be done using a 1st order Taylor polynomial and still be different from Gradient Descent?
These sites are causing me to question:


*

*Univariate Newton's Method http://fourier.eng.hmc.edu/e176/lectures/NM/node20.html

*Multivariate Gradient Descent  http://fourier.eng.hmc.edu/e176/lectures/NM/node27.html
 A: Like in the comments stated; gradient descent and Newton's method are optimization methods, independently if its univariate or multivariate. Gradient descent only uses the first derivative, which sometimes makes it less efficient in multidimensional problems because Newton's method attracts to saddle points. Newton's method uses the curvature of the function (the second derivative) which lead generally faster to a solution if the second derivative is easy to compute. So they can both be used for multivariate and univariate optimization, but the performance will generally not be similar.
A: Let us assume that we are trying to minimize $f(x)$ w.r.t $x$, with $f$ differentiable.
Let $\nabla f(x) = f'(x)$ be the first derivative of $f(x)$ w.r.t $x$, and $\nabla^2 f(x) = f'(f'(x)) = f''(x)$ be the second derivative of $f(x)$ w.r.t $x$.
To minimize $f(x)$ w.r.t $x$,  we can use either the Gradient Descent method or Newton's Method, both of them are iterative methods.
$\textbf{Gradient Descent method}$ has the form:
$$ x^{t+1} = x^{t} - \eta (\nabla f(x))$$
where $\eta $ is a step size parameter. Here we use the first only. Thus, it is called a 1st-order optimization method.
$\textbf{Newton's Method}$
$$ x^{t+1} = x^{t} - \eta (\frac{\nabla f(x)}{\nabla^2 f(x)}) $$
here we use the first derivative and the second derivative, thus this method is called a 2nd-order optimization method.
