A proof of the the second derivative test? Suppose $f\in C^3$ in some ball centered at a, where $a\in \Bbb{R}^2$,and $\nabla f=0$ at a, but not all second derivatives of $f$ are zero at a. Show how can local maximums local minimums or neither can be determnined from the Taylor polynomial of $f$ at a (of degree 2) .
How does this extend to $\Bbb{R}^n$? Does the method still work if $f\in C^n?$
I think this is related to the hessian matrix, but I dont know how to prove it. And why does the question say the polynomial is of degree 2?  
 A: When $f:\ {\mathbb R}^n\to{\mathbb R}$ is in $C^2$ and $\nabla f(a)=0$ then according to Taylor's theorem
$$f(a+X)-f(a)=h(X)+o(|X|^2)\qquad(X\to0)\ .$$
Here $X=(X_1,\ldots, X_n)$ is a displacement vector attached at $a$, and
$$h(X)=\sum_{i,k} h_{ik}X_i X_k,\qquad h_{ik}:={\partial^2 f\over \partial x_i\partial x_k}\biggr|_a\ ,$$
is the so-called Hessian of $f$ at the critical point $a$.
When this quadratic form $h$ is positive definite then it assumes a  minimum $\mu>0$ on the unit sphere $S^{n-1}\subset \Bbb R^n$, and it follows that
$$f(a+X)-f(a)\geq |X|^2\bigl(\mu +o(1)\bigr)\qquad(X\to0)\ .$$
It follows that $f(a+X)-f(a)>0$ for all sufficiently small $|X|$, so that we can conclude that $f$ assumes a local minimum at $a$. Similarly, when $h$ is negative definite, the function $f$ assumes a local maximum at $a$.
When $h$ is indefinite there is a point $U\in S^{n-1}$ with $h(U)<0$ and a point $V\in S^{n-1}$ with $h(V)>0$. Looking at displacement vectors $X$ of the form $X=t U$, resp. $X=t V$ and arguing as above one sees that $f(a+X)-f(a)$ assumes as well positive as negative values suitable $X$ near $0$. Therefore we will not have a local extremum of $f$ at $a$ in this case.
In the cases considered so far the determinant $\det h:=\det[h_{ik}]$ was nonzero. When this determinant happens to be zero the critical point $a$ is called degenerate. In this case it is in general difficult to find out whether one has a local extremum. Consider as an example the function
$$f(x,y):=(y-x^2)(y-2x^2)\ .$$
One easily computes $\det h(0,0)=0$, so that the above second-derivative-test is inconclusive. Now in this simple example we can see directly what happens:
Between the two parabolas $y=x^2$ and $y=2x^2$ the function $f$ is negative, but otherwise positive. Therefore the function $f$ does not assume a local extremum at $O=(0,0)$, even though along any line through $O$ the function has a local minimum at $O$.
