For which values of $a\in \mathbb{R}$ is there a max or min for $f(x,y) = (x-1)^2 +2a(x-1)y +y^2 +y^4$ in $(1,0)$? Given $a \in \mathbb{R}$, and $$f(x,y)=(x-1)^2 +2a(x-1)y + y^2+y^4$$ determine which values for $a$ that will make $f(x,y)$ have a maximum or minimum in $(1,0)$.
My attempt so far is as follows:
I calculate the partial derivatives
$f_{x} = 2ay + 2(x-1)$, $f_{y} = 4y^3 +2y +2a(x-1)$, $f_{xx} = 2$ , $f_{yy} = 12y^2 +2$, $f_{xy} = 2a$.
And aquire the charateristic polynomial for the point $(1,0)$:
$$Q(h,k)= 2h^2 + 4ahk + 2k^2$$
From here I am unsure about how to determine which values for $a$ that will make the polynomial tell me the point is a max or min.
Another approach, which I am less sure about being correct is using the Hessian matrix.
It becomes
$$\begin{bmatrix} 2a & 2\\\
2 & 2a\end{bmatrix}$$
which has the determinant $4a^2-4$ which is $>0$ for all $a>1$ and $a<-1$. 
Is this approach correct?
Please answer both how to do it using the characteristic polynomial and if the method with the Hessian matrix is correct.
 A: A critical point is an extremum if the Hessian is definite (all eigenvalues have the same sign) and a saddle point if both positive and negative eigenvalues are present. If one or more eigenvalues are zero, and the others are all the same sign, then the Hessian test fails.
Since we are in 2 dimensions, there are only two eigenvalues. If they are of the same sign, then their product - the determinant - will be positive. If both signs are present, the determinant will be negative. And if the determinant is zero, then the test fails. This simplification of looking at the determinant only works in 2 dimensions.
Since the determinant is $4 - 4a^2$, you need $|a| < 1$ to have an extremum. But for completeness, you also need to investigate when $a = \pm 1$. It is possible for the Hessian to be $0$ at an extremum or saddle point.

To have a maximum means that if you leave it in any direction, the value goes down. To have a minimum means that to leave it in any direction, the value goes up. So we want to know what the value of $f$ is doing as we move away from $(1,0)$ in any direction. We choose a direction $(h,k)$ and head out on the line $(1,0) + t(h,k)$. We can assume that $h^2 + k^2 = 1$, since any other value just multiplies $t$ by some constant. Define $g(t) = f(1 + th, tk)$. Then $$\frac {dg}{dt} = h\frac{\partial f}{\partial x}(1+th, tk) + k \frac{\partial f}{\partial y}(1+th, tk)$$
$$\frac {d^2g}{dt^2} = h^2\frac{\partial^2 f}{\partial x^2}(1+th, tk) + 2hk\frac{\partial^2 f}{\partial x\partial y}(1+th, tk) + k \frac{\partial^2 f}{\partial y^2}(1+th, tk)$$
Setting $t = 0$, we see that $\frac {dg}{dt} = 0$, so $0$ is a critical point of $g$. We use the 2nd derivative test to see if it is an extremum or inflection point. At $t = 0$, we find that $$\frac {d^2g}{dt^2} = 2h^2 + 4ahk + 2k^2 = Q(h,k)$$
That is, the second derivative of $g$ is the characteristic polynomial. But now we know what it means. If $\frac {d^2g}{dt^2} > 0$ in every direction, this must be a minimum. If $\frac {d^2g}{dt^2} < 0$ in every direction, this must be a maximum.
If there are directions where $\frac {d^2g}{dt^2} < 0$  and others where $\frac {d^2g}{dt^2} > 0$, it must be a saddle point, because leaving the point in some directions will take you up, while other directions will take you down. If there is at least one direction where $\frac {d^2g}{dt^2} = 0$ and every non-zero direction has the same sign, then this test fails.
So can we choose $h, k$, not both $0$, to make $Q(h, k) > 0$, to make it $< 0$ or to make it $ = 0$? That can be hard to figure out at first glance. But as I already noted, it is sufficient to look only at $h, k$ with $h^2 + k^2 = 1$, which allows us to set $h = \cos \theta, k = \sin \theta$. Then the polynomial becomes
$$Q(\cos \theta, \sin \theta) = 2\cos^2 \theta + 2\sin^2 \theta + 2a\sin 2\theta = 2(1 + a\sin 2\theta)$$
Are there values of $\theta$ where $Q > 0$, values where $Q < 0$, values where $Q = 0$?
Obviously, 


*

*When $|a| < 1$, $Q > 0$ always. In this case $(1,0)$ is a minimum.

*When $|a| > 1$, there are values of $\theta$ with $Q < 0$ and other values with $Q > 0$. $(0,1)$ is a saddle point.

*When $|a| = 1$, Then $Q \ge 0$, but there are values of $\theta$ where $Q$ is $0$, so the test fails.



Can you figure out what happens when $|a| = 1$?
