"Gradient descent is a first-order optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point."

71 questions
Filter by
Sorted by
Tagged with
5k views

### Gradient of a function as the direction of steepest ascent/descent

I am trying to really understand why the gradient of a function gives the direction of steepest ascent intuitively. Assuming that the function is differentiable at the point in question, a) I had a ...
• 492
42k views

In order to find the local minima of a scalar function $p(x), x\in \mathbb{R}^3$, I know we can use the gradient descent method: $$x_{k+1}=x_k-\alpha_k \nabla_xp(x)$$ where $\alpha_k$ is the step size ...
• 5,268
10k views

### Does gradient descent converge to a minimum-norm solution in least-squares problems?

Consider running gradient descent (GD) on the following optimization problem: $$\arg\min_{\mathbf x \in \mathbb R^n} \| A\mathbf x-\mathbf b \|_2^2$$ where $\mathbf b$ lies in the column space of $A$...
• 3,145
885 views

### A matrix calculus problem in backpropagation encountered when studying Deep Learning

I am studying the Algorithm 6.4 in the textbook Deep Learning, which is about backpropagation. I am confused by this line: $$\nabla_{W^{(k)}}J = gh^{(k-1)T}+\lambda\nabla_{W^{(k)}}{\Omega(\theta)}$$ ...
• 885
47k views

### What is the difference between projected gradient descent and ordinary gradient descent?

I just read about projected gradient descent but I did not see the intuition to use Projected one instead of normal gradient descent. Would you tell me the reason and preferable situations of ...
• 1,127
8k views

### Why is Newton's method faster than gradient descent?

Can you provide some intuition as to why Newton's method is faster than gradient descent? Often we are in a scenario where we want to minimize a function f(x) where x is a vector of parameters. To do ...
• 1,074
45k views

### Optimal step size in gradient descent

Suppose a differentiable, convex function $F(x)$ exists. Then $b = a - \gamma\nabla F(a)$ implies that $F(b) \leq F(a)$ given $\gamma$ is chosen properly. The goal is to find the optimal $\gamma$ at ...
• 1,567
10k views

• 17.7k
9k views

### Gradient is NOT the direction that points to the minimum or maximum

I understand that the gradient is the direction of steepest descent (ref: Why is gradient the direction of steepest ascent? and Gradient of a function as the direction of steepest ascent/descent). ...
• 426
705 views

My question is very similar to this one and this one, but they haven't been answered. Let $f \in C^2(\mathbb{R}^d, \mathbb{R})$ have compact sublevel sets and isolated critical points, and consider ...
• 1,722
486 views

### Gradient descent: step size for a $C^{\infty}$ coercive function

Let $f: U \to \mathbb R$ be a $C^{\infty}$ function where $U$ is an open connected subset of $\mathbb R^n$. $f$ is coercive, i.e., $f(x) \to +\infty$ as $\|x\| \to \partial U$. This is equivalent to ...
• 3,548
193 views

### is it wrong? -> using Gradient to get the steepest slope to go upwards, so in order get minimized loss we go the opposite [closed]

I am not good at mathematics and have been learning ML from Udacity. In its tutorial video, the tutor says(I concluded it in a short way): using Gradient to get the steepest slope to go upwards, so in ...
• 119
216 views

### Is the directional derivative $f'(x;d)$ ALWAYS equal to $\nabla f^T d$? or only under certain conditions?

My professor always writes the directional derivative with the condition that the functionis $C^1$ smooth. i see him write many times the directional derivative written as $f'(x;d)=\nabla f^T d$, but ...
• 1,603
91 views

### Why is Gradient Descent always chosen for Neural Networks?

I am trying to understand why Gradient Descent are the chosen types of algorithm for optimizing the Loss Function in Neural Networks - and why other algorithms (e.g. EM Algorithm https://en.wikipedia....
• 3,224
50k views

### What is the difference between the Jacobian, Hessian and the Gradient?

I know there is a lot of topic regarding this on the internet, and trust me, I've googled it. But things are getting more and more confused for me. From my understanding, The gradient is the slope of ...
• 951
23k views

• 25.5k
3k views

### Intuition for gradient descent with Nesterov momentum

A clear article on Nesterov’s Accelerated Gradient Descent (S. Bubeck, April 2013) says The intuition behind the algorithm is quite difficult to grasp, and unfortunately the analysis will not be ...
• 921
712 views

### A possible bug in a highly cited paper (Adam gradient descent)?

My question comes from the paper Adam: A Method for Stochastic Optimization published on ICLR, which has been cited over $80,000$ times up to now. Specifically, in page 13, I think there is a 'bug' in ...
• 2,860
625 views

### Physical interpretation of gradient descent

Introduction Here are some high-level intuitions that seem to be folklore in the optimization community: The gradient descent method is often motivated from a physical point of view, as a 'ball ...
• 3,560
5k views

### Why Does the Projected Gradient Descent Method Work?

Consider the problem \begin{align*} \min_{x \in \mathbb{R}^n} &\quad f(x) \\ s.t.: &\quad x \in C, \end{align*} where $C$ is a convex set. As $C$ is convex, the projection onto $C$, $P_C$, is ...
• 95
2k views

### Trying to understand the math behind backpropagation in neural nets

I am currently trying to understand the math used training neural network, in which gradient descent is used to minimize the error between the target and extracted. I currently following/reading this ...
• 325
346 views

### Why must a function be closed for descent methods to work?

At the beginning of chapter 9 of Boyd & Vandenberghe's Convex Optimization, about unconstrained minimization, it is said: The starting point $x^{(0)}$ for a method must lie in $\mathbf{dom} f$, ...
701 views

• 3,548
285 views

• 411
210 views

### Gradient of Coefficient of Variation of weighted geometric mean

This question is another version of the one here which I've previously asked. the difference is instead of WM here we have exp(WM). exp(W×log(M)) is equivalent to weighted geometric mean of M when W ...
• 165
594 views

### How to find Minimum of Function with Many Local Minima

I'm trying to find the smallest local minima within a given boundary condition, e.g., a circle. Currently, I'm using a monte carlo algorithm to approximate the minimum, followed by a gradient descent ...
• 355
322 views

### first term in the asymptotic expansion using method of steepest descent

I am working with the following intgral: $\int_{0}^{\infty}t^{n}e^{-x(t+\frac{1}{t})}dt$ as $x\rightarrow \infty$ Now, I have been trying to solve this using the method of steepest descent. After ...
• 423