# Questions tagged [gradient-descent]

"Gradient descent is a first-order optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point."

403 questions
9 views

16 views

### Confused about Nesterov momentum gradient descent algorithm

I've found a variety of variations of writing Nesterov but I cannot understand why they cannot simply be expanded into a one liner. Here is one I found that can just be re-arranged, can someone ...
22 views

### Quadratic Gradient Descent Optimum Step Size

First i have searched this forum but could not find a question that matched mine, though some are somewhat similar. my issue is whether or not the signage matters when i try to calculate the optimum ...
72 views

### Using Coordinate Descent on Projected Space

My goal is to maximize an objective function using coordinate descent over a 3-dimensional vector. In the simple case the domain over which I am maximizing is defined as follows: $X \in \mathcal{X}$ ...
8 views

### `Antisymmetric Preconditioning' for Gradient Descent

When minimising a convex function $f : \mathbf{R}^d \to \mathbf{R}$, a standard approach is to work with the gradient flow ODE \begin{align} \dot{x} = - \nabla f (x) \end{align} and then take some ...
36 views

### Gradient of Quadratic Form with Inverse of Complex Matrices

I want to calculate the gradient of $$w^H H F (F^H F)^{-1} F^H H^H w$$ with respect to $F$, which is complex. I am basing on this previous answer Derivative of Nested Matrix Quadratic Form ...
21 views

### Zoutendijk's Lemma Using Goldstein Conditions

I am reading Numerical Optimization by Wright and Nocedal and in page 39, it says that a similar result to Zoutendijk's lemma (Theorem 3.2) can be proven using the Goldstein conditions instead of the ...
32 views

18 views

### Why each component of gradient which is slope of the curve in itself while keeping other variables constant gives us slope of curve?

My doubt is suppose we assume a 3D space with 2D surface in it given by some function z = f(x,y). Then each component of the gradient is geometrically the slope of the tangent at f on either x-z or y-...
15 views

16 views

### Additive Gradient Descent with negative weights error tends to be maximized (MSE) — solved

Suppose you have a cost function $C(x) = \frac{1}{2}(y - a)^2$ where $y$ is the desired output and $a$ is an activation. There is only one training example of $x = 1$ where the desired output $y = -5$...
57 views

### Slow convergence of gradient descent for a strictly convex quadratic

Let $0 < \lambda_1 \leq \lambda_2 \leq \ldots \leq \lambda_n$ and let $f: \mathbb{R}^n \to \mathbb{R}$ define by $$f(x) = \frac{1}{2}x^TMx$$ where M is \begin{bmatrix} \lambda_1 & 0 &...
33 views

### gradient for sum on 100 variables analytically

I need to minimize a function with x in $R^{100}$ and $a_i$ is a given vector. The function itself is: $\sum_{i=1}^{500} log(1- a_i^t x) - \sum_{i=1}^{100} log(1-x_i^2)$ The first thing I thought ...
45 views

### Finding Lowest Elevation Path Between Two Points

Let's say I have a matrix of values that represent heights with function $f(x,y)$ and I am trying to find the "lowest value path" beween two points. So this would be the reverse of hill climbing, as ...
44 views

### Why my gradient descent seems to diverge “pair-wise”?

Why my gradient descent seems to diverge "pair-wise"? I've checked the algorithms and they work for golden section line search and "small step parameter". However, when trying to get the algo to ...
55 views

35 views

### How to take derivative of log loss function in gradient descent?

I know the gradient descent about $z=wx+b$. But how to implement the derivative values of $w$ and $b$ in Python? I see some example like ...
23 views

### Projected Conjugate Gradient or BFGS for bound constrained optimization

We know how projected gradient descent works for bound constrained optimization (https://neos-guide.org/content/gradient-projection-methods). It is basically steepest descent with an additional ...
66 views

### Gradient Descent for Exponential Functions

I am trying to develop a non-linear regression for several functions (power, log and exponential). the idea was to use a log transformation to get an initial set of points, close enough to the real ...