How to determine the intersection of a plane and a cone? I want to know what is the intersection of the plane $2x+y+z-2=0$ and $x^2+y^2=z^2$. My guess is that it is an ellipse or an hyperbola, but I can't understand how to determine which one.
Subsituting $z=2-2x-y$ in the cone equation, we obtain
$$
5 x^2 + 4 x y - 8 x + 2 y^2 - 4 y + 4 = 0
$$
and I can't undertsand whether it is an hyperbola or an ellipse.
Will you please help me figure it out? 
Thanks
 A: Making the change of variables ( a rigid rotation does not alter the form)
$$
x = X\cos\theta-Y\sin\theta\\
y = X\sin\theta+Y\cos\theta
$$
in
$$
f(x,y) = 5 x^2 + 4 x y - 8 x + 2 y^2 - 4 y + 4 = 0
$$
we get at
$$
f(x,y)\equiv F(X,Y) = \frac{1}{2} (4 \sin (2 \theta)+3 \cos (2 \theta)+7)X^2+(4 \cos (2 \theta)-3 \sin (2 \theta))X Y+\frac{1}{2} (7-4 \sin (2 \theta)-3 \cos (2 \theta))Y^2-12\cos\theta X+12\sin\theta Y+4=0
$$
and now choosing $\theta$ such that $4\cos(2\theta)-3\sin(2\theta) = 0\to \theta = \frac 12 \arctan(\frac 43)$ we obtain finally
$$
F(X,Y) =6 X^2-\frac{24 X}{\sqrt{5}}+Y^2+\frac{12 Y}{\sqrt{5}}+4 = 0
$$
which is clearly an ellipse. 
NOTE
As can be verified the intersection of the cone $x^2+y^2=z^2$ and the plane $2x+y+z-2=0$ gives
$$
f(x,y) = 4 - 8 x + 3 x^2 - 4 y + 4 x y = 0
$$
and not 
$$
f(x,y) = 5 x^2 + 4 x y - 8 x + 2 y^2 - 4 y + 4 = 0
$$
as assumed. In this case with the same rotation the true intersection curve should be
$$
F(X,Y) = 4 X^2-4 \sqrt{5} X-Y^2+4
$$
which is a hyperbola.
A: You have $$
\tag{*} 5 x^2 + 4 x y - 8 x + 2 y^2 - 4 y + 4 = 0
$$
and your goal is to figure out whether that is an ellipse or a hyperbola. We can rephrase that question as, "is it possible that the equation can keep having solutions for large coordinate values?"
For large sizes of the coordinates, the quadratic terms
$$ 5x^2 + 4xy + 2y^2 $$
will dominate, but it is not immediately clear whether they can keep canceling out each other, because the sign of $4xy$ can be both positive and negative. But we can get rid of the mixed term by completing the square together with one of the other terms -- for example,
$$ 2(x+y)^2 = 2x^2 + 4xy + 2y^2 $$
so
$$ 5x^2 + 4xy + 2y^2 = 5x^2 + 2(y+x)^2 - 2x^2 = 3x^2 + 2(x+y)^2 $$
which is large and positive for large coordinate values and therefore cannot be canceled out by the linear terms which grow slower.
So this shape must be an ellipse (or a point, or nothing at all).

Unfortunately, though, your equation (*) is not right.
On geometric grounds, if the intersection is an ellipse, then $(0,0)$, which is on the axis of the cone, ought to be inside the ellipse. However, the LHS of (*) is positive at $(0,0)$, and we have just found that it is also positive when the coordinates are large. That doesn't make sense. So something must be wrong.
It looks like you inserted your $z=2-2x-y$ into $x^2+y^2+z^2=0$ instead of into $x^2+y^2=z^2$. So now you need to go back and do something like the above for the equation you get after fixing this sign error.
A: There was probably a typo error or calculation error in coefficient of $x^2$. It should be
$$ 3 x^2 + 4 x y - 8 x + 2 y^2 - 4 y + 4 = 0 $$
which as others also pointed out, by considering discriminant $ ab -h^2 <0 $ should be a hyperbola.
A: Let me offer a more geometrical approach. To check if the intersection is an ellipse, a parabola or a hyperbola it is enough to check whether the plane intersects all the generatrices of the cone or not. In practice, this can be done as follows.


*

*Construct the vector $\vec n$ perpendicular to the plane; in your case you can read it off the equation of the plane: $\vec n=(2,1,1)$.

*Construct a vector $\vec t$ perpendicular to $\vec n$ and such that plane $(\vec n,\vec t)$ is parallel to the axis of the cone. If $\vec l$ is any vector along the axis, then a suitable choice is 
$$\displaystyle\vec t={\vec n\cdot\vec n\over\vec n\cdot\vec l}\ \vec l-\vec n.$$
In your case you can take $\vec l=(0,0,1)$ to get $\vec t=(-2,-1,5)$.

*Check if point $P=V+\vec t$ is inside the cone or not, where $V$ is the cone vertex. In your case $V=(0,0,0)$ and $P=(-2,-1,5)$; if $P$ is inside the cone its coordinates should satisfy $x^2+y^2<z^2$, as it is indeed the case.

*If $P$ is inside the cone, then the intersection is a hyperbola (your case); if $P$ lies on the cone surface, then the intersection is a parabola; if $P$ is outside the cone, then the intersection is an ellipse.
