Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gradient-descent]

"Gradient descent is a first-order optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point."

0
votes
0answers
9 views

Derivative (Jacobian) of transposed function

Let $x \in R^n$, $F \in R^{m \times n}$ and $f(x) = Fx$. It's easy to conclude that the Jacobian of $f(x)$ is $Df(x) = F$. Where $Df(x)_{ij} = \frac{\partial f_i}{\partial x_j}$. Therefore $\nabla ...
3
votes
1answer
44 views

Textbook Recommendations: Solving Systems of Matrix ODEs

This is in reference to the works of Trendafilov whose approaches to multivariate statistical problems boil down to solving a dynamical system involving matrices. Question: Can anyone suggest a book/...
0
votes
1answer
20 views

Proof of Batch Gradient Descent's cost function gradient vector

In the book Hands-On Machine Learning with Scikit-Learn & TensorFlow, the author only showed the formula for the Batch Gradient Descent method, such as: $ \dfrac{\partial}{\partial \theta_{j}} ...
0
votes
0answers
16 views

Confused about Nesterov momentum gradient descent algorithm

I've found a variety of variations of writing Nesterov but I cannot understand why they cannot simply be expanded into a one liner. Here is one I found that can just be re-arranged, can someone ...
1
vote
1answer
22 views

Quadratic Gradient Descent Optimum Step Size

First i have searched this forum but could not find a question that matched mine, though some are somewhat similar. my issue is whether or not the signage matters when i try to calculate the optimum ...
2
votes
0answers
72 views

Using Coordinate Descent on Projected Space

My goal is to maximize an objective function using coordinate descent over a 3-dimensional vector. In the simple case the domain over which I am maximizing is defined as follows: $X \in \mathcal{X}$ ...
0
votes
0answers
8 views

`Antisymmetric Preconditioning' for Gradient Descent

When minimising a convex function $f : \mathbf{R}^d \to \mathbf{R}$, a standard approach is to work with the gradient flow ODE \begin{align} \dot{x} = - \nabla f (x) \end{align} and then take some ...
1
vote
1answer
36 views

Gradient of Quadratic Form with Inverse of Complex Matrices

I want to calculate the gradient of $$ w^H H F (F^H F)^{-1} F^H H^H w $$ with respect to $ F $, which is complex. I am basing on this previous answer Derivative of Nested Matrix Quadratic Form ...
0
votes
0answers
21 views

Zoutendijk's Lemma Using Goldstein Conditions

I am reading Numerical Optimization by Wright and Nocedal and in page 39, it says that a similar result to Zoutendijk's lemma (Theorem 3.2) can be proven using the Goldstein conditions instead of the ...
0
votes
1answer
32 views

Getting to the gradient descent algorithm

I understand that gradient descent comes from the (quite natural) idea that we might want to choose our next weight vector ($w^{t+1}$) as $$w^{t+1} = \arg \min_w \frac{1}{2} \|w-w^t\|^{2} + \eta f(w^...
0
votes
0answers
33 views

I am trying to find the maximum learning rate or stepping rate of steepest descent algorithm in 2 dimensions

Let $f(x,y) = (x-y)^4+2x^2+y^2-x+2y$. I am trying to numerically find the miniumum of $f$. We define a fixed-point iteration scheme \begin{align*} g(x, y) = \vec{x}_{n+1} = \vec{x}_{n} - a\nabla f \...
1
vote
0answers
29 views

Are the initial values of the slope, intercept and the learning rate in gradient descent experimental?

I am having some problems trying to implement a gradient descent algorithm. Like I said in the title, initial values of these properties drastically change the outcome of the slope and the intercept ...
0
votes
0answers
14 views

Richardson's Iteration, Gradient Method and Spectral Radius

Richardson's iteration introduce a scalar $\alpha$ to the update formula: $$ \textbf{x}^{(k+1)} = \textbf{x}^{(k)} + \alpha \textbf{r}^{(k)} $$ And compute $\alpha$ by minimizing the spectral radius:...
1
vote
1answer
42 views

How do you calculate gradient descent when you have a point which gives two different values?

I am trying to implement gradient decent algorithm. The dataset on which I am working has points which are of a partial function I guess. For example these are a subset of the dataset. $(1, 10)$ $(...
0
votes
0answers
17 views

Classical gradient descent optimization of smooth nonnegative function $f$ restricted to hypercube $C = [-R,R]^s$.

Let $f : \mathbb{R}^s \rightarrow \mathbb{R}_{\geq 0}$ be a smooth, nonnegative function and $R > 0$. Now chose a point $x_0 \in (-R,R)^s$ in the interior of the compact hypercube $C := [-R,R]^s$. ...
0
votes
0answers
9 views

Expand a discrete 2D function

I have discrete 2D function $Z$ defined over grid $X \times Y$, where X and Y numbers grid's collumns and rows. Generating values of $Z$, however, is very costly, preventing me from reaching ...
0
votes
0answers
11 views

Reason for differences in EWA equations and Momentum equations

I am trying to learn more about optimization algorithms regularly used in deep learning, starting with alternating least squares and gradient descent. One snag I have run into is a difference in ...
0
votes
0answers
59 views

Projected gradient descent with matrices

I have a scalar function $f(\rho) = Tr(\rho H) + c\ Tr(\rho\log\rho)$, where $Tr$ is trace, $\rho$ is a positive semidefinite matrix with trace 1 and $H$ is a Hermitian matrix and $c$ is a postive ...
1
vote
0answers
28 views

Derivation of partial derivative of cost function with respect to weights in backpropagation algorithm

I am studying Machine Learning from Andrew Ng's Machine Learning course on coursera. I am stuck at understanding math behind back propagation. Here is an image of backpropagation algorithm from his ...
2
votes
1answer
51 views

Mazur's lemma without Hahn-Banach theorem/axiom of choice?

In the development of gradient-flow theory (in Hilbert-space $H$), we soon stumble on the question whether the function $u \mapsto \varphi[u]:=\frac{1}{2}\|u\|^2+I[u]$ -where $I:H \to \mathbb{C}$ is ...
1
vote
0answers
46 views

Velocity gradient in polar coordinate

I just want to convert velocity gradient in polar coordinate to velocity gradient in Cartesian one (i.e. $\frac{\partial u_r}{\partial r}=f\left(\frac{\partial v}{\partial y}\right)$). How can I ...
0
votes
1answer
41 views

Matrix calculus - incorrect calculation?

I understand that matrix derivatives involve different possible conventions but even with that in mind, I'm not sure how to show the result I have. It is given according to this calculation that $$\...
0
votes
1answer
18 views

Why each component of gradient which is slope of the curve in itself while keeping other variables constant gives us slope of curve?

My doubt is suppose we assume a 3D space with 2D surface in it given by some function z = f(x,y). Then each component of the gradient is geometrically the slope of the tangent at f on either x-z or y-...
0
votes
1answer
15 views

For any starting point $x^{(0)}$ the basic conjugate algorithm converges to the unique $x^{*}$ in $n$ steps

Proof I would like to check my understanding about the above. Thoughts So for this we would have $Q,x^{(0)},b$ and we're trying to find $x^{(*)}$. The set of Q-conjugate vectors $d^{(i)}, \; i \in ...
0
votes
0answers
19 views

Coordinate-wise gradient descent converges to least-squares solution

Does somebody know a reference (or maybe short proof/argument) for the following claim: Coordinate-wise gradient descent converges to a least-squares solution. Coordinate-wise gradient descent: ...
4
votes
1answer
45 views

Directional derivative confusion - why does independently evaluating partial changes, then adding them, work?

I apologize for both my crude math grammar, and what is probably an obvious question - I am a novice. I am confused as to why, when taking the directional derivative, the gradient is evaluated by ...
0
votes
0answers
14 views

How to take derivative of multivariate Taylor series matrices?

Suppose we consider a function $f(x)$ with $f(x):\mathbb{R}^n \to \mathbb{R}$. We let $r(x)$ be the second order Taylor series of $f(x)$ about the base point $z \in \mathbb{R}^n$. How can I show ...
0
votes
1answer
35 views

How to update the weights knowing the loss in Neural Network

Question How update the weight knowing the loss Work explanation I have a really simple network composed by two layers with one neurons in both layers. Considering an input of 1, the final result ...
1
vote
1answer
44 views

Gradient of an interpolated function

Can anyone please give a explanation: what do you mean by gradient of an interpolated function? Suppose, $f(x, y, z) = 2x^3 + 3y^2 -z$ is a function, and one result of the interpolation for the ...
1
vote
1answer
24 views

Alternative ways to write the gradient

This may not be the right forum to ask this question, but suppose that I have a multi-dimensional function $L$, and I want to compute its gradient w.r.t. a set of parameters $\theta$, where $[|\theta| ...
1
vote
0answers
16 views

Additive Gradient Descent with negative weights error tends to be maximized (MSE) — solved

Suppose you have a cost function $C(x) = \frac{1}{2}(y - a)^2$ where $y$ is the desired output and $a$ is an activation. There is only one training example of $x = 1$ where the desired output $y = -5$...
1
vote
3answers
57 views

Slow convergence of gradient descent for a strictly convex quadratic

Let $0 < \lambda_1 \leq \lambda_2 \leq \ldots \leq \lambda_n$ and let $f: \mathbb{R}^n \to \mathbb{R}$ define by $$ f(x) = \frac{1}{2}x^TMx $$ where M is \begin{bmatrix} \lambda_1 & 0 &...
0
votes
0answers
33 views

gradient for sum on 100 variables analytically

I need to minimize a function with x in $R^{100}$ and $a_i$ is a given vector. The function itself is: $\sum_{i=1}^{500} log(1- a_i^t x) - \sum_{i=1}^{100} log(1-x_i^2) $ The first thing I thought ...
1
vote
2answers
45 views

Finding Lowest Elevation Path Between Two Points

Let's say I have a matrix of values that represent heights with function $f(x,y)$ and I am trying to find the "lowest value path" beween two points. So this would be the reverse of hill climbing, as ...
0
votes
0answers
44 views

Why my gradient descent seems to diverge “pair-wise”?

Why my gradient descent seems to diverge "pair-wise"? I've checked the algorithms and they work for golden section line search and "small step parameter". However, when trying to get the algo to ...
1
vote
2answers
55 views

Proximal gradient method justification

If $f$ and $g$ are respectively a differentiable function and a convex, lower semi-continuous function, then the algorithm defined by: $$ x^{k+1} = \text{prox}_{\gamma{g}}[x^{k} - \gamma\nabla{f(x^{k}...
1
vote
1answer
67 views

Calculate gradient of the spectral norm analytically

Given a matrix $F \in \mathbb{C}^{m \times n}$ such that a $m>n$ and other matrix $A$ (non-symmetric matrix) of size $n \times n$ and spectral norm as: $$\|A-F^*\operatorname{diag}(b)F\|_2 = \...
0
votes
0answers
35 views

How to take derivative of log loss function in gradient descent?

I know the gradient descent about $z=wx+b$. But how to implement the derivative values of $w$ and $b$ in Python? I see some example like ...
1
vote
0answers
23 views

Projected Conjugate Gradient or BFGS for bound constrained optimization

We know how projected gradient descent works for bound constrained optimization (https://neos-guide.org/content/gradient-projection-methods). It is basically steepest descent with an additional ...
0
votes
0answers
66 views

Gradient Descent for Exponential Functions

I am trying to develop a non-linear regression for several functions (power, log and exponential). the idea was to use a log transformation to get an initial set of points, close enough to the real ...
1
vote
0answers
46 views

Complex scalar derivative of trace of complex matrices $\frac{d}{d z} Tr[A U(z) B U^H(z)] $

I'm trying to numerically find the maximum of $$ f(z) = Tr[ A\;U B\; U^H], \quad U=U(z,z^*),\\ A,B,U\in\mathbb{C}^{n\times n},\;\; A=A^H, \quad B=B^H $$ using gradient descent(ascent) w.r.t. $z \in {...
0
votes
0answers
18 views

Problem in calculating the gradient of a function?

I am trying to understand the gradient calculation of a formula, which is an optimization function and trying to maximize the GDL (Generalized Dice Loss) where, w_l...
0
votes
0answers
15 views

Will the gradient ascent method work?

I'm trying to use the gradient ascent method on a function $F{\theta}$, where $F$ is the multiplication of Normal multivariate densities, with a composition of a very complex function of the ...
2
votes
1answer
43 views

Looking for two separate functions with intersecting point and equal gradient

I want to explain a disadvantage of Gradient Descent where the gradient itself doesn't give information about how far we are away from the local/global minimum. Say we have two functions with an ...
0
votes
1answer
19 views

Explanation of gradient descent on convex quadratic

Can someone explain the following: $$f(x) = \frac{1}{2}w^TAw - b^Tw$$ Assume AA is symmetric and invertible, then the optimal solution $w^{\star}$ occurs at $$w^{\star} = A^{-1}b$$ and $$\nabla f(w)...
0
votes
0answers
38 views

Newton's Method for a step size to move in the direction of the gradient

I am reading this article that talks about Newton's method that can give us an ideal step size to move in the direction of the gradient. I do not understand what $\epsilon$ is in the following part ...
0
votes
0answers
47 views

How relevant are theoretical convex optimization convergence rates in practice, when parameters are unknown and function may be nonconvex?

There are many theoretical results known on convergence rates for various (possibly stochastic) convex optimization problems. For example, the popular review on optimization algorithms for machine ...
0
votes
1answer
45 views

Convergence rate of Gradient Descent

I was trying to solve a simple gradient descent problem. If we have $f(x) = x^2$, and a learning rate, $\eta$, that guarantees that the algorithm converges, then in how many steps will my algorithm be ...
0
votes
0answers
8 views

Gradient ascent curves

Let $f:R^2\rightarrow R$ be continuously differentiable $m$ times. Pick $(x,y) \in R^2$. Define the "gradient ascent curve from $(x,y)$ on $f$" to be the limit of the path followed by performing ...
0
votes
0answers
21 views

Total Least Square fitting

Say I want to fit a straight line using Total Least Square (as opposed to Least Square), which is to minimize the sum of (yi-k*xi-b)^2/(k^2+1) over all xi's and yi's, where xi's and yi's are training ...