# Questions tagged [numerical-optimization]

Numerical methods for continuous optimization.

909 questions
Filter by
Sorted by
Tagged with
72k views

### Partial derivative in gradient descent for two variables

I've started taking an online machine learning class, and the first learning algorithm that we are going to be using is a form of linear regression using gradient descent. I don't have much of a ...
21k views

In order to find the local minima of a scalar function $p(x), x\in \mathbb{R}^3$, I know we can use the gradient descent method: $$x_{k+1}=x_k-\alpha_k \nabla_xp(x)$$ where $\alpha_k$ is the step size ...
14k views

### What is the difference between projected gradient descent and ordinary gradient descent?

I just read about projected gradient descent but I did not see the intuition to use Projected one instead of normal gradient descent. Would you tell me the reason and preferable situations of ...
17k views

### Optimal step size in gradient descent

Suppose a differentiable, convex function $F(x)$ exists. Then $b = a - \gamma\nabla F(a)$ implies that $F(b) \leq F(a)$ given $\gamma$ is chosen properly. The goal is to find the optimal $\gamma$ at ...
9k views

### What is the definition of a first order method?

The term "first order method" is often used to categorize a numerical optimization method for say constrained minimization, and similarly for a "second order method". I wonder what is the exact ...
355 views

### Gradient descent for differentiable convex functions

Suppose $f\colon\mathbb{R}^n\to\mathbb{R}$ is convex and differentiable, and assume that $f$ has a minimizer. If $(x_k)$ is the sequence generated by exact gradient descent, must it converge to a ...
560 views

### A numerical optimization problem with a convolution in the constraint

I have a problem of the following form: minimize $\|Dx\|_2$ subject to $\|x*x\|_2 = 1$ where $x\in\mathbb R^n$, $D$ is a given diagonal matrix of positive entries, and $*$ represents convolution, ...
7k views

### Why does gradient descent work?

On Wikipedia, this is the following description of gradient descent: Gradient descent is based on the observation that if the multivariable function $F(\mathbf{x})$ is defined and differentiable ...
2k views

### What Numerical Methods Are Known to Solve ${L}_{1}$ Regularized Quadratic Programming Problems?

What numerical methods are suitable to solve the following problem $$\min_x \tfrac{1}{2}x^T A x + b^Tx + \lambda ||x||_1$$ where $x,b\in\mathbf{R}^n$, and $A\in \mathbf{R}^{n\times n}$ is positive ...
677 views

4k views

### High Dimensional Optimization Algorithm?

I have an optimization problem that at first sounds quite textbook. I have a convex objective function in $D$-dimensional space that is twice differentiable everywhere and has no local optima. ...
2k views

### Good Textbook in Numerical PDEs?

I am currently taking a course on Numerical PDE. The course covers the following topics listed below. Chapter 1: Solutions to Partial Dierential Equations: Chapter 2: Introduction to Finite ...
3k views