22
$\begingroup$

In my classes, we are taught the following.

If the determinant of the Hessian matrix at the critical point $\det(D^2f(c)) > 0$ and $f_{xx}(c) > 0$, the function $f$ at c is concave up.

If the determinant of the Hessian matrix at the critical point $\det(D^2f(c)) > 0$ and $f_{xx}(c) < 0$, the function $f$ at c is concave down.

If the determinant of the Hessian matrix at the critical point $\det(D^2f(c)) < 0$, the function $f$ at c is a saddle point.

However, the reasoning behind this is never explained. We are never taught WHY or HOW.

I would like to know why the determinant of the Hessian matrix, combined with the second derivative at the critical point, contains this information about max., min., and saddle points. I would also like to know how this is derived, as I think this would likely go hand-in-hand with why.

Please give clear reasoning behind each step - not just 'this is what it is' without any reasoning.

When researching this topic, I recall reading mentions regarding eigenvectors or eigenvalues, but I honestly cannot remember.

Thank you.

$\endgroup$
3
  • $\begingroup$ Do you know what eigenvalues and eigenvectors actually are, or have you merely seen them mentioned? $\endgroup$
    – Ian
    Oct 26, 2016 at 12:36
  • $\begingroup$ @Ian I have just begun studying them. I previously saw them mentioned as being connected to this theory behind the Hessian matrix. $\endgroup$ Oct 26, 2016 at 12:38
  • 1
    $\begingroup$ OK. They're central to the way that this subject actually works. $\endgroup$
    – Ian
    Oct 26, 2016 at 12:39

3 Answers 3

41
$\begingroup$

Given a smooth function $f: \mathbb{R}^n \to \mathbb{R}$, we can write a second order Taylor expansion in the form: $$f(x + \Delta x) = f(x) + (\nabla f(x))^\top \Delta x + \frac{1}{2}( \Delta x)^\top Hf(x) \Delta x + O(|\Delta x|^3)$$ where $\nabla f (x)$ is the gradient of $f$ at $x$, $Hf(x)$ is the Hessian matrix of $f$ at $x$, and $\Delta x$ is some small displacement. Note that since the Hessian matrix is symmetric, then $n$ (posibly repeated) real eigenvalues, and in particular it is diagonalizable as a real matrix.

Suppose you have have a critical point at $x=a$, then $\nabla f(a)= 0$. Then your Taylor expansion looks like: $$f(a + \Delta x) = f(a) + \frac{1}{2}( \Delta x)^\top Hf(a) \Delta x + O(|\Delta x|^3).$$ Thus, for small displacements $\Delta x$, the Hessian tells us how the function behaves around the critical point.

  • The Hessian $Hf(a)$ is positive definite if and only if $( \Delta x)^\top Hf(a) \Delta x > 0 $ for $\Delta x \neq 0$. Equivalently, this is true if and only if all the eigenvalues of $Hf(a)$ are positive. Then no matter which direction you move away from the critical point, the value of $f(a + \Delta x)$ grows (for small $|\Delta x|$), so $a$ is a local minimum.

  • Likewise, the Hessian $Hf(a)$ is negative definite if and only if $( \Delta x)^\top Hf(a) \Delta x < 0 $ for $\Delta x \neq 0$. Equivalently, this is true if and only if all the eigenvalues of $Hf(a)$ are negative. Then no matter which direction you move away from the critical point, the value of $f(a + \Delta x)$ decreases (for small $|\Delta x|$), so $a$ is a local maximum.

  • The Hessian $Hf(a)$ is indefinite if and only if it has at least one positive and at least one negative eigenvalue. Then $f(a + \Delta x)$ either increases or decreases depending on the direction you move away from the critical point so $a$ is a saddle point.

  • Finally, the test fails if we have a zero eigenvalue but the rest of the eigenvalues are either all larger than or smaller than $0$. In this case we need to use higher order terms to decide if we have a saddle or local max/min.

What I've described for you here is the intuition for the general situation on $\mathbb{R}^n$, but since it seems like you're working in $\mathbb{R}^2$, the test becomes a bit simpler. In $\mathbb{R}^2$ we can only have two (possibly identical) eigenvalues $\lambda_1$ and $\lambda_2$ for $Hf(a)$, since it is a symmetric $2 \times 2$ matrix. We can take advantage of the fact that the determinant of a matrix is the product of the eigenvalues, and the trace is their sum: $\det(Hf(a))=\lambda_1 \lambda_2$ and $\operatorname{tr}(Hf(a))=\lambda_1 + \lambda_2$.

In this situation:

  1. $\det (Hf(a))=0$ means that there is a zero eigenvalue and so the test fails.

  2. $\det(Hf(a))<0$ means that both eigenvalues have different sign, so we have a saddle point at $a$.

  3. $\det(Hf(a))>0$ means that both eigenvalues have the same sign: either both positive or both negative, and we must use the trace to decide which it is. In fact, rather than use the trace, it actually suffices to just use the top left entry $\frac{\partial^2 f}{\partial x^2} (a)$ of $Hf(a)$ to decide, by Sylvester's criterion. In other words, $\frac{\partial^2 f}{\partial x^2}(a) > 0$ means both eigenvalues are positive (local min at a $a$), whereas $\frac{\partial^2 f}{\partial x^2} (a) < 0$ means both eigenvalues are negative (local max at $a$).

$\endgroup$
4
  • $\begingroup$ Thanks. What is the $O(|\Delta x|^3)$ term? I'm not sure what the $O$ is supposed to represent. $\endgroup$ Oct 26, 2016 at 17:04
  • 1
    $\begingroup$ It's big O notation. It's not a great notation, in my opinion, but it tells you rough size of the error in the second order Taylor approximation. $\endgroup$
    – ಠ_ಠ
    Oct 26, 2016 at 20:27
  • $\begingroup$ At a saddle point, there are directions where $f$ neither increases nor decreases to second order. Consider $xy$ on the axes. Please check your third and fourth bullet points. A zero eigenvalue is not the same thing as having $v^t$ orthogonal to $H v$. The latter is guaranteed at saddle points. $\endgroup$ Dec 10, 2017 at 6:15
  • 1
    $\begingroup$ I'm assuming that $( \Delta x)^t$ is the transpose of the displacement vector $( \Delta x)$? It just confused me for a little bit, so I thought it should be mentioned somewhere here that that's what it is. $\endgroup$
    – Mr. Nichan
    Jul 15, 2020 at 14:12
7
$\begingroup$

It rests on the multivariable Taylor's formula: $$f(x)=f(c)+Df(c)\cdot(x-c)+\frac1{2!}(D^2f)_{x=c}(x-c)+o(\lVert x-c\rVert^2)$$ For a critical point, $Df(c)=0$, and the formula can be written as $$f(x)-f(c)=\frac1{2!}(D^2f)_{x=c}(x-c)+o(\lVert x-c\rVert^2)$$

where $(D^2f)_{x=c}$ denotes the quadratic form associated with the hessian matrix at $x=c$.

When $\lVert x-c\rVert$ is small enough and the quadratic form is not $0$, the sign of this expression is the sign of the quadratic form. Hence if the quadratic form is positive, $f(x)-f(c)>0$, corresponding to $f(c)$ being a local minimum, if it is negative, it corresponds to a local maximum.

For two variables, setting $x-c=(h,k)$, the quadratic form is, explicitly: $$\frac{\partial^2f}{\partial x^2}(c)h^2+2\frac{\partial^2f}{\partial x\partial y}(c)hk+\frac{\partial^2f}{\partial y^2}(c)k^2$$ This is a homogeneous quadratic form in two variables, and its sign is the same as the sign of the dehomogenised quadratic form in one variable (setting $t= h/k$): $$\frac{\partial^2f}{\partial x^2}(c)t^2+2\frac{\partial^2f}{\partial x\partial y}(c)t+\frac{\partial^2f}{\partial y^2}(c)$$ Now a non-zero quadratic polynomial has constant sign if and only if its (reduced) discriminant is negative, whence the condition for an extremum: $$\biggl[\biggl(\frac{\partial^2f}{\partial x^2}\biggr)^2-\frac{\partial^2f}{\partial x\partial y}\biggr](c)<0.$$ Furthermore, under these circumstances, the sign of the quadratic polynomial is the sign of its leading coefficient.Hence it is

  • positive (local minimum) if $\dfrac{\partial^2f}{\partial x^2}(c)<0$,
  • negative (local maximum) if $\dfrac{\partial^2f}{\partial x^2}(c)>0$.
$\endgroup$
3
  • $\begingroup$ $D^2 f (x-c)^2$ is kind of a weird way to write it, since we don't multiply vectors. I'd prefer to write either $(x-c)^T D^2 f (x-c)$ (thinking of $D^2 f$ as a matrix) or $D^2 f(x-c,x-c)$ (thinking of $D^2 f$ as a tensor, i.e. a multilinear scalar-valued function). $\endgroup$
    – Ian
    Oct 26, 2016 at 12:38
  • $\begingroup$ @Ian: I agree it's weird, but I specified it's a quadratic form. I tried to use a compact notation, but the problem is the (c). Finally I changed it to $(D^2f)_{x=c}(x-c)$, which is mathematically correct. $\endgroup$
    – Bernard
    Oct 26, 2016 at 12:51
  • 1
    $\begingroup$ I think you meant to write $[\frac{\partial^2 f}{\partial x \partial y} - \frac{\partial^2 f}{\partial x^2} \frac{\partial^2 f}{\partial y^2}](c) < 0$, no? Since that's the reduced discriminant of the polynomial of variable $t$? Amazing answer, btw! $\endgroup$
    – Striker
    Aug 19, 2018 at 20:18
2
$\begingroup$

You can see it in this way. Determinant is the product of all eigenvalues of the Hessian matrix (2 eigenvalues, in the case of two variables). Then checking the sign of determinant is sufficient to tell the sign of eigenvalues, which is a more general way to test the min/max points.

FYI: wiki.

$\endgroup$
1
  • $\begingroup$ I don't understand this point. If you have a positive determinant of the Hessian, that doesn't mean all of your eigenvalues were positive. They could all be negative and the negatives multiply out to positives. Please explain further. $\endgroup$
    – Galen
    Feb 20, 2023 at 1:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .