# Questions tagged [numerical-optimization]

Numerical methods for continuous optimization.

909 questions
Filter by
Sorted by
Tagged with
4answers
72k views

### Partial derivative in gradient descent for two variables

I've started taking an online machine learning class, and the first learning algorithm that we are going to be using is a form of linear regression using gradient descent. I don't have much of a ...
4answers
21k views

### Gradient descent with constraints

In order to find the local minima of a scalar function $p(x), x\in \mathbb{R}^3$, I know we can use the gradient descent method: $$x_{k+1}=x_k-\alpha_k \nabla_xp(x)$$ where $\alpha_k$ is the step size ...
1answer
14k views

### What is the difference between projected gradient descent and ordinary gradient descent?

I just read about projected gradient descent but I did not see the intuition to use Projected one instead of normal gradient descent. Would you tell me the reason and preferable situations of ...
3answers
17k views

### Optimal step size in gradient descent

Suppose a differentiable, convex function $F(x)$ exists. Then $b = a - \gamma\nabla F(a)$ implies that $F(b) \leq F(a)$ given $\gamma$ is chosen properly. The goal is to find the optimal $\gamma$ at ...
3answers
9k views

### What is the definition of a first order method?

The term "first order method" is often used to categorize a numerical optimization method for say constrained minimization, and similarly for a "second order method". I wonder what is the exact ...
2answers
355 views

### Gradient descent for differentiable convex functions

Suppose $f\colon\mathbb{R}^n\to\mathbb{R}$ is convex and differentiable, and assume that $f$ has a minimizer. If $(x_k)$ is the sequence generated by exact gradient descent, must it converge to a ...
2answers
560 views

### A numerical optimization problem with a convolution in the constraint

I have a problem of the following form: minimize $\|Dx\|_2$ subject to $\|x*x\|_2 = 1$ where $x\in\mathbb R^n$, $D$ is a given diagonal matrix of positive entries, and $*$ represents convolution, ...
3answers
7k views

### Why does gradient descent work?

On Wikipedia, this is the following description of gradient descent: Gradient descent is based on the observation that if the multivariable function $F(\mathbf{x})$ is defined and differentiable ...
2answers
2k views

### What Numerical Methods Are Known to Solve ${L}_{1}$ Regularized Quadratic Programming Problems?

What numerical methods are suitable to solve the following problem $$\min_x \tfrac{1}{2}x^T A x + b^Tx + \lambda ||x||_1$$ where $x,b\in\mathbf{R}^n$, and $A\in \mathbf{R}^{n\times n}$ is positive ...
1answer
677 views

3answers
4k views

### High Dimensional Optimization Algorithm?

I have an optimization problem that at first sounds quite textbook. I have a convex objective function in $D$-dimensional space that is twice differentiable everywhere and has no local optima. ...
2answers
2k views

### Good Textbook in Numerical PDEs?

I am currently taking a course on Numerical PDE. The course covers the following topics listed below. Chapter 1: Solutions to Partial Dierential Equations: Chapter 2: Introduction to Finite ...
1answer
3k views

1answer
5k views

### Gauss-Newton versus gradient descent

I would like to ask first if the second order gradient descent method is the same as the Gauss-Newton method. There is something I didn't understand. I read that with the Newton's method the step we ...
1answer
389 views

### What is the purpose of an oracle in optimization?

I am reading a textbook on convex optimization, and in it there was an extremely short discussion of so called "oracle model" on page 136, which has left me confused. Pardon my ignorance but why do ...