"Gradient descent is a first-order optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point."

880 questions
Filter by
Sorted by
Tagged with
1 vote
49 views

### why gradient descent does not always land at the global minimum closest to the starting point?

I am given this function $\boxed{f(x,y)=((x^2+y^2)-1)^2}$. I need to do gradient descent analysis on it. I have studied that it's not trivial to show mathematically "ball reaches to the global ...
11 views

### Gradient calculation for an autoregressive probit model

I am using the following specification to estimate a binary choice Probit model: $$P(y_t=1|x_t) = \Phi(\pi_t),$$ where $\Phi$ is the cumulative distribution function of the normal distribution. My ...
15 views

### Taking multiple optimization steps on the same trajectory not well justified.

I am reading the paper Proximal Policy Optimization Algorithms found at https://arxiv.org/pdf/1707.06347.pdf. In this paper they say "While it is appealing to perform multiple steps of ...
1 vote
15 views

I would like to know how to compute the gradient of Function Defined by Integral For example, Let $\displaystyle f(x(t),y(t))=\int_{0}^{k}g(x(t),y(t))\rm{d}t$ be function and $t\in\mathbb{R}^{+}$ $$\... 0 votes 0 answers 9 views ### What is the relation between strong convexity and the step-sizes in gradient descent? I would like to use gradient descent methods for minimising a strong convexity function where$$f(x)-f(\bar{x})-\nabla f(\bar{x})^T(x-\bar{x}) \geq \frac{m}{2}\|x-\bar{x}\|^2$$I would like to know ... 0 votes 0 answers 28 views ### Proving the bound on the expectation of stochastic gradient descent conditional on previous step I'm trying to do a question that asks show that E[f(Xk+1)|Xk] ≤ f(Xk), k ≥ 0. where f is an L-smooth, convex function that is the average of fi(x). Stochastic gradient descent is applied to start ... 0 votes 0 answers 34 views ### multiple losses with L-BFGS in torch I am using the torch iimplementation of L-BFGS to minimize a loss function, which is given as the sum of n loss function loss = loss_1 + loss_2 + ... + loss_n Computing each of the losses loss_i is ... 1 vote 4 answers 71 views ### Question About Gradient Descent Gradient descent is numerical optimization method for finding local/global minimum of function. It is given by following formula:$$ x_{n+1} = x_n - \alpha \nabla f(x_n) $$For sake of simplicity let ... 0 votes 1 answer 93 views ### What Exactly is Step Size in Gradient Descent Method? Gradient descent is numerical optimization method for finding local/global minimum of function. It is given by following formula:$$ x_{n+1} = x_n - \alpha \nabla f(x_n) $$There is countless content ... 0 votes 0 answers 19 views ### Understanding the rewritten form of Nesterov Accelerated Gradient used to derive Nadam This, other papers and this blog suggest that we can rewrite the NAG algorithm which basically does$$\theta _{t+1\:}=\theta _t\:-m_t$$with$$m_t=\gamma \:m_{t-1}+\eta \frac{\partial L(\theta _t-\...
Say for $x \in \mathbb{R}^n$, I'm minimizing $\|Ax - b \|_2^2$ with L1 regularization on selected entries of $x$. i.e. instead of directly add a $\|x\|_1$ regularization term, it would be on \$|x_i| + |...