# Questions tagged [numerical-optimization]

Numerical methods for continuous optimization.

892 questions
Filter by
Sorted by
Tagged with
72k views

### Partial derivative in gradient descent for two variables

I've started taking an online machine learning class, and the first learning algorithm that we are going to be using is a form of linear regression using gradient descent. I don't have much of a ...
20k views

In order to find the local minima of a scalar function $p(x), x\in \mathbb{R}^3$, I know we can use the gradient descent method: $$x_{k+1}=x_k-\alpha_k \nabla_xp(x)$$ where $\alpha_k$ is the step size ...
13k views

### What is the difference between projected gradient descent and ordinary gradient descent?

I just read about projected gradient descent but I did not see the intuition to use Projected one instead of normal gradient descent. Would you tell me the reason and preferable situations of ...
17k views

### Optimal step size in gradient descent

Suppose a differentiable, convex function $F(x)$ exists. Then $b = a - \gamma\nabla F(a)$ implies that $F(b) \leq F(a)$ given $\gamma$ is chosen properly. The goal is to find the optimal $\gamma$ at ...
9k views

### What is the definition of a first order method?

The term "first order method" is often used to categorize a numerical optimization method for say constrained minimization, and similarly for a "second order method". I wonder what is the exact ...
345 views

### Gradient descent for differentiable convex functions

Suppose $f\colon\mathbb{R}^n\to\mathbb{R}$ is convex and differentiable, and assume that $f$ has a minimizer. If $(x_k)$ is the sequence generated by exact gradient descent, must it converge to a ...
553 views

### A numerical optimization problem with a convolution in the constraint

I have a problem of the following form: minimize $\|Dx\|_2$ subject to $\|x*x\|_2 = 1$ where $x\in\mathbb R^n$, $D$ is a given diagonal matrix of positive entries, and $*$ represents convolution, ...
On Wikipedia, this is the following description of gradient descent: Gradient descent is based on the observation that if the multivariable function $F(\mathbf{x})$ is defined and differentiable ...