# Why does the Hessian work?

I am working through Susskind's 'The Theoretical Minimum' (on physics) – it also includes some maths. In particular, there is an interlude for which he discusses partial differentiation. He discusses a surface, a function of two variable.

Let $A=f(x,y)$

\begin{pmatrix} \frac{\partial^2A}{\partial x^2} & \frac{\partial^2A}{\partial x \partial y} \\ \frac{\partial^2A}{\partial y \partial x} & \frac{\partial^2A}{\partial y^2} \end{pmatrix}

I understand individually what the partial derivatives are doing (though shouldn't the second and third entries into the matrix be equivalent?). I can also accept that they can be arranged into a matrix.

However, he then says

If the determinant and the trace are positive, the point is a local minimum.

If the determinant is positive and the trace negative, the point is a local maximum.

If the determinant is negative, the point is a saddle point.

I'm sure I could apply these rules, but why do these rules work?

• The second and third are only equivalent if the function is $C^2$ on an open disk around the point (IE the second partials are all continuous in a region around the area you're working in)
– Alan
Jul 24, 2015 at 0:29
• The point is to look at the Taylor expansion of $f$ around such point. Try to find a proof of the theorem you're citing and you'll understand how it works.
– Pedro
Jul 24, 2015 at 0:36
• That's almost exactly the same as how $f''(x)$ works.
– Vim
Jul 24, 2015 at 1:30
• From one-dimensional calculus the first derivative set equal to zero finds the minimum points of the function. The second derivative then tells wether the function evaluated at that point is a minimum, maximum, or saddle point. Conceptually extend this to a two dimensional function, which defines a surface, and apply the just mentioned tests. Aug 2, 2015 at 2:27

For some reason, the geometric meaning of the Hessian is rarely made explicit. If $f\colon \mathbb{R}^n\to\mathbb{R}$ is a $C^2$ function in a neighborhood of a point $p$, the Hessian matrix $Hf_p$ is the matrix for the quadratic form that is the second directional derivative of $f$ at $p$. What this means is that, if $\textbf{v}$ is an $n$-dimensional vector, then $\textbf{v}^T(Hf_p)\textbf{v}$ is the directional second derivative of $f$ at $p$, i.e. $$\textbf{v}^T(Hf_p)\textbf{v} \;=\; \frac{d^2}{dt^2}\bigl[f(p+t\textbf{v})\bigr]\biggr|_{t=0}$$ This equation can be derived fairly easily using the multivariable chain rule.

For a critical point $p$, the directional second derivatives usually determine whether $p$ is a local minimum, a local maximum, or a saddle point. In particular:

• If the directional second derivative is positive in every direction, then $p$ is a local minimum.

• If the directional second derivative is negative in every direction, then $p$ is a local maximum.

• If the directional second derivative is positive in some directions and negative in other directions, then $p$ is a saddle point.

These statements follow from the usual second-derivative test in single-variable calculus, where the single variable functions in question are the cross-sectional functions $t\mapsto f(p+t\textbf{v})$.

All of this relates to certain facts about symmetric matrices from linear algebra:

• A symmetric matrix $A$ has the property that $\textbf{v}^TA\textbf{v} > 0$ for all nonzero vectors $\textbf{v}$ if and only if $A$ has all positive eigenvalues. (Such a matrix is called positive definite.)

• A symmetric matrix $A$ has the property that $\textbf{v}^TA\textbf{v} < 0$ for all nonzero vectors $\textbf{v}$ if and only if $A$ has all negative eigenvalues. (Such a matrix is called negative definite.)

• A symmetric matrix $A$ satisfies $\textbf{v}^TA\textbf{v} > 0$ for some vectors and $\textbf{v}^TA\textbf{v} < 0$ for other vectors if and only if it has at least one positive eigenvalue and at least one negative eigenvalue.

Applying these facts to the Hessian gives:

• If $p$ is a critical point for $f$ and $Hf_p$ is positive definite, then $p$ is a local minimum for $f$.

• If $p$ is a critical point for $f$ and $Hf_p$ is negative definite, then $p$ is a local maximum for $f$.

• If $p$ is a critical point for $f$ and $Hf_p$ has at least one positive eigenvalue and at least one negative eigenvalue, then $p$ is a saddle point for $f$.

For a $2\times 2$ matrix, you can determine the signs of the eigenvalues by investigating the trace and the determinant. This is because the trace of a matrix is the sum of its eigenvalues, and the determinant of a matrix is the product of its eigenvalues. In particular:

• A $2\times 2$ matrix has two positive eigenvalues if and only if the trace and determinant are both positive

• A $2\times 2$ matrix has two negative eigenvalues if and only if the trace is negative and the determinant is positive.

• A $2\times 2$ matrix has one positive and one negative eigenvalue if and only if the determinant is negative.

Note that these statements don't hold for $3\times 3$ or larger matrices. For example, a $3\times 3$ matrix with eigenvalues $-2,-1,10$ will have positive trace ($7$) and positive determinant ($20$). For such a matrix, you really have to determine the eigenvalues explicitly, or use something like Sylvester's criterion to determine whether the Hessian is positive definite, negative definite, or neither.

Edit: Since this seems to be my main post about the Hessian, I should mention that the Hessian can also be interpreted as the matrix for a symmetric bilinear form: $$(\mathbf{v},\mathbf{w}) \,\mapsto\, \mathbf{v}^T(Hf_p)\mathbf{w}.$$ This bilinear form represents the "mixed" second directional derivative of $f$ in the directions of $\mathbf{v}$ and $\mathbf{w}$. That is, if $D_{\mathbf{v}}f$ represents the directional derivative of $f$ in the direction of $\mathbf{v}$, then $$\mathbf{v}^T(Hf)\mathbf{w} \,=\, D_{\mathbf{v}}D_{\mathbf{w}}f = D_{\mathbf{w}}D_{\mathbf{v}}f.$$ Equivalently, $$\mathbf{v}^T(Hf_p)\mathbf{w} \,=\, \frac{\partial^2}{\partial s\,\partial t}\bigl[f(p+s\mathbf{v}+t\mathbf{w})\bigr]\biggr|_{s,t=0}.$$ When $\mathbf{v}=\mathbf{w}$ this reduces the to second directional derivative in the direction of $\mathbf{v}$ described above.

I think it would be more clear if he described the three cases as: 1) the Hessian is positive definite; 2) the Hessian is negative definite; 3) the Hessian has one positive eigenvalue and one negative eigenvalue.

The key idea here is that the second order Taylor approximation to $f$ at $(x,y)$ tells us how $f$ behaves near $(x,y)$. When $\nabla f(x,y) = 0$, the Hessian term in the Taylor approximation tells us how $f$ behaves near $(x,y)$.