What is the higher-order derivative test in multivariable calculus? In single-variable calculus, the second-derivative test states that if $x$ is a real number such that $f'(x)=0$, then:


*

*If $f''(x)>0$, then $f$ has a local minimum at $x$.  

*If $f''(x)<0$, then $f$ has a local maximum at $x$. 

*If $f''(x)=0$, then the text is inconclusive.


But there's no need to despair if the second-derivative test is inconclusive, because there is the higher-order derivative test.  It states that if $x$ is a real number such that $f'(x)=0$, and $n$ is the smallest natural number such that $f^{(n)}(x)\neq 0$, then:


*

*If $n$ is even and $f^{(n)}>0$, then $f$ has a local minimum at $x$.

*If $n$ is even and $f^{(n)}<0$, then $f$ has a local manimum at $x$.

*If $n$ is odd, then $f$ has an inflection point at $x$.


Similarly, in multivariable calculus the second-derivative test states that if $(x,y)$  is an ordered pair such that $\nabla f(x,y) = 0$, then:


*

*If $D(x,y)>0$ and $f_{xx}(x,y)>0$, then $f$ has a local minimum at $(x,y)$.

*If $D(x,y)>0$ and $f_{xx}(x,y)<0$, then $f$ has a local maximum at $(x,y)$.

*If $D(x,y)<0$, then $f$ has a saddle point at $(x,y)$.

*If $D(x,y)=0$, then the test is inconclusive.


where $D(x,y)=f_{xx}(x,y)f_{yy}(x,y)-(f_{xy}(x,y))^2$ is the determinant of the Hessian matrix of $f$ evaluated at $(x,y)$.
My question is, what do you do if this test is inconclusive?  What is the analogue of the higher-order derivative test in multivariable calculus?
 A: This webpage states and proves a version of the higher-order derivative test that applies not only to functions defined on $\mathbb{R}^2$ or $\mathbb{R}^N$, but functions defined on arbitrary Banach spaces.  First there is this theorem:

Theorem 38 (Higher derivative test). Let $A\subseteq E$ be an open set and let f$:A\to\mathbb{R}$. Assume that $f$ is $(p-1)$ times continuously differentiable and that $D^p f(x)$ exists for some $p\ge 2$ and $x\in A$. Also assume that $f'(x),\dots,f^{(p-1)}(x)=0$ and $f^{(p)}(x)\ne 0$. Write $h^{(p)}$ for the $p$-tuple $(h,\dots,h)$.

*
  
*If $f$ has an extreme value at $x$, then $p$ is even and the form $f^{(p)}(x)h^{(p)}$ is semidefinite.
  
*If there is a constant $c$ such that $f^{(p)}(x)h^{(p)}\ge c > 0$ for all $|h|=1$, then $f$ has a strict local minimum at $x$ and (1) applies.
  
*If there is a constant $c$ such that $f^{(p)}(x)h^{(p)}\le c < 0$ for all $|h|=1$, then $f$ has a strict local maximum at $x$ and (1) applies.
  
  

Then there is this corollary for the finite dimensional case, which is what we’re interested in:

Corollary 39 (Higher derivative test, finite-dimensional case). In Theorem 38, further assume that $E$ is finite-dimensional. Then $h\mapsto f^{(p)}(x)h^{(p)}$ has both a minimum and maximum value on the set $\{h\in E:|h|=1\}$, and:

*
  
*If the form $f^{(p)}(x)h^{(p)}$ is indefinite, then $f$ does not have an extreme value at $x$.
  
*If the form $f^{(p)}(x)h^{(p)}$ is positive definite, then $f$ has a strict local minimum at $x$.
  
*If the form $f^{(p)}(x)h^{(p)}$ is negative definite, then $f$ has a strict local maximum at $x$.
  
  

Here $f^{(p)}(x)$ denotes a tensor containing all the pure and mixed partial derivatives of $f$ of order $p$, evaluated at $x$.
A: Consider a homogeneous polynomial $f(x_1, \ldots, x_n)$ of
total degree $d > 0$ in $n$ variables.  In order to tell that $(0,\ldots,0)$ is a 
local minimum, we would need to know that $f(x_1,\ldots,x_n) \ge 0$ for all $x_1,\ldots,x_n$.  Unfortunately this is a difficult problem in general, and
I'm pretty sure there are no very simple tests, although there are algorithms   related to Hilbert's 17th problem.
A: In that case if the test is inconclusive there is not a general rule or test that always work. We need case be case to show what kind of critical point we have using some manipulation and inequalities.
As a simple example for
$$f(x,y)=x^4-2x^2y^2+y^4$$
at $(x,y)=(0,0)$ the test is inconclusive but
$$f(x,y)=x^4-2x^2y^2+y^4=(x^2-y^2)^2\ge 0$$
