Skip to main content

# Questions tagged [gradient-descent]

"Gradient descent is a first-order optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point."

1,041 questions
Filter by
Sorted by
Tagged with
0 votes
0 answers
26 views

### Derivation of the upper bound of the average regret of online-to-batch conversion in H-smoothness

I've been studying a paper (Smoothness, Low-Noise and Fast Rates) on the impact of smoothness on the convergence rate of online-to-batch conversion, specifically Theorem 2, which provides a bound on ...
• 129
1 vote
0 answers
48 views

### Using differentiation under the integral sign for computing the gradient of the expectation.

I am following the work from Kingma and Welling, where they introduce Variational Autoencoders. To train such models they use the so called Evidence Lower Bound and maximize it. One way to solve this ...
0 votes
1 answer
53 views

2 votes
0 answers
34 views

• 81
1 vote
0 answers
70 views

• 1,052
0 votes
0 answers
13 views

### Help understand a gradient derivation for RankNET

I am reading the RankNet to LambdaMART paper : https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/MSR-TR-2010-82.pdf , where the author makes a particular claim in equation (1). They ...
• 894
0 votes
0 answers
19 views

### How to get the gradient of this function?

I'm trying to approximate a image with a sum of radial basis functions $\phi(x_i, \theta_i)$ where each $x_i\in R²$ and it is the coordinate of every pixel of the original image. $\theta_i$ is the ...
0 votes
0 answers
29 views

### Computing gradient over all examples in Gradient Descent

I am studying about Gradient Descent and Stochastic Gradient Descent, and the text says that one of the advantages of sgd over gd is, that gd can be computationally expensive for large datasets. In ...
0 votes
0 answers
54 views

### Gradient of a complex scalar with respect to a real vector

I have a question regarding a regression problem. Suppose vector $\mathbf{p}$ is real and has length N. Now suppose $\mathbf{p}$ is the input to the following equation: $$y = f(e^{i\mathbf{p}})$$ ...
0 votes
0 answers
22 views

### Relationship between model variance and dataset size for SGD

I'm looking for some function that describes the variance of model weights when trained with Stochastic Gradient Descent for $m$ independent minibatches. I can apply central limit theorem to a single ...
1 vote
1 answer
49 views

0 votes
0 answers
16 views

### What is the steady state distribution of this Poisson process with non-constant rate?

I am looking for the steady state distribution of the following Poisson process: $$d x(t) = -k_1(x(t)-k_2)dt + k_3dN(t)$$ where $k_1$, $k_2$ and $k_3$ are constants and the rate $\lambda(x)$ of the ...
0 votes
0 answers
12 views

### Results on convergence and runtime rates of proximal algorithms

Are there any known results on the convergence rates and computational runtime of proximal algorithms? I'm interested in finding out how well they scale with increasing number of input dimensions, but ...
• 119
2 votes
1 answer
69 views

### Gradient of Function of Complex Matrices

Consider the following function: $$f(T) = \| T^{T}TB - C\|^2_2$$ where $T, B,$ and $C$ are all complex matrices. Let $T = X + iY.$ I wish to compute $\nabla f$ i.e. $\dfrac{\partial f}{\partial T}$....
5 votes
1 answer
141 views

### Discretization Error of Mirror Descent

It is well known that for sufficiently differentiable functions $f$ and small $\eta>0$ the iterate given by gradient descent $$x_{k+1}=x_k-\eta \nabla f(x_k)$$ is within $\mathcal O(\eta^2)$ of ...
• 2,337
1 vote
1 answer
81 views

• 179