Given four points, determine a condition on a fifth point such that the conic containing all of them is an ellipse The image of the question if you don't see all the symbols
The given points $p_1,p_2,p_3,p_4$ are located at the vertices of a convex quadrilateral on the real affine plane. 
I am looking for an explicit condition on the point $p_5$ necessary and sufficient for the conic which is determined by $p_1,p_2,p_3,p_4,p_5$ to be an ellipse. 
Could you give me a hint, please? 
I tried to "go" to $\mathbb P_2$ and change the coordinates of these points to more convenient (for example, if $a=(1:a_1,a_2)$ change it to $(1:1:0)$) but I am not sure that this transformation preserves conic
 A: I'll expand-upon my hint, because the raw calculations get pretty ugly, yet there is a reasonably-tame way to express the desired condition.

The equation of a conic through $P$, $Q$, $R$, $S$, $T$ is given by:
$$\left|\begin{array}{cccccc} x^2 & y^2 & x y & x & y & 1 \\ P_x^2 & P_y^2 & P_x P_y & P_x & P_y & 1 \\ Q_x^2 & Q_y^2 & Q_x Q_y & Q_x & Q_y & 1 \\ R_x^2 & R_y^2 & R_x R_y & R_x & R_y & 1 \\ S_x^2 & S_y^2 & S_x S_y & S_x & S_y & 1 \\ T_x^2 & T_y^2 & T_x T_y & T_x & T_y & 1 \\ \end{array} \right| = 0 \tag{1}$$
Expanding the determinant yields an equation of the form
$$A x^2 + B x y + C y^2 + D x + E y + F = 0 \tag{2}$$
This represents an ellipse when
$$B^2-4AC < 0 \tag{3}$$
(likewise, a hyperbola when $>0$, and a parabola when $=0$).
Condition $(3)$, in $xy$-coordinates, turns out to be an expression in over $14,000$ terms. We can collapse the complexity a bit by using a self-induced coordinate system; specifically, we'll use barycentric coordinates based on $\triangle PQR$ (which we'll assume is non-degenerate).
We can give $S$ and $T$ respective coordinates $(s_P:s_Q:s_R)$ and $(t_P:t_Q:t_R)$. That is, we can write
$$S = \frac{s_P P + s_Q Q + s_R R}{s_P+s_Q+s_R} \qquad\qquad
T = \frac{t_P P + t_Q Q + t_R R}{t_P + t_Q+t_R} \tag{4}$$
Substitution into $(3)$ collapses the relation into a mere $21$ terms (and a discardable factor corresponding to the area of $\triangle PQR$). This is better, but still a little messy. It cleans up nicely when reciprocating the elements, however; defining $x' := 1/x$, we can write

$$\begin{align}
&\phantom{4}\left(
  s^{\prime}_P t^{\prime}_Q
+ s^{\prime}_Q t^{\prime}_R
+ s^{\prime}_R t^{\prime}_P
+ s^{\prime}_Q t^{\prime}_P
+ s^{\prime}_R t^{\prime}_Q 
+ s^{\prime}_P t^{\prime}_R \right)^2 \\[4pt]
<\; &4 \left(s^{\prime}_P s^{\prime}_Q + s^{\prime}_Q s^{\prime}_R + s^{\prime}_R s^{\prime}_P \right) \left(t^{\prime}_P t^{\prime}_Q + t^{\prime}_Q t^{\prime}_R + t^{\prime}_R t^{\prime}_P \right) \\
\left(\quad =\; \right. & \frac{4}{s_P s_Q s_R\,t_P t_Q t_R}\left.\left(s_P+s_Q+s_R\right)\left(t_P+t_Q+t_R\right)\quad\right)
\end{align} \tag{$\star$}$$

If we know $P$, $Q$, $R$, $S$ (and therefore $s_P$, $s_Q$, $s_R$ and $s^\prime_P$, $s^\prime_Q$, $s^\prime_R$), then $(\star)$ gives a condition on parameters defining $T$.
(There's probably a nice projective interpretation of $(\star)$ that could've saved all the trouble of deriving it. :)
A: Adding to the direct answer
in terms of parabolas:
If you are reduced to ruler and compass
or just do not want to actually draw the two bounding parabolas,
you may want to transform the problem to a geometrically simpler one.
The method described below does this.
It can be extended to find slopes of asymptotes (if any) and of principal axes
when given five points of a conic section.
To simplify the description, I assume a projective geometry context.
Thus I may speak of points at infinity and the line at infinity.
Points at infinity occur as intersections of parallel lines and
can be interpreted as slopes of those lines.
A point is at infinity if and only if it lies on the line at infinity.
Relabel $p_1,\ldots,p_4$ to $A,B,C,D$. Any permutation is admissible.
(Some details will depend on the permutation, but the overall result won't.)
Let me also rename $p_5$ to $E$.
We will use $ABC$ as reference triangle below.
Relative to that reference triangle $ABC$, there is a transformation
of points called isogonal conjugation.
The inverse transformation is again isogonal conjugation.
However, isogonal conjugation of the vertices $A,B,C$ is undefined.
Therefore isogonal conjugation is a bijection only
for points outside the lines $\overline{AB}$, $\overline{BC}$, $\overline{CA}$.
In our use case, the resulting singularities can be removed by continuity.
Finding the isogonal conjugate of a point can be done using only ruler and
compass. Furthermore:

*

*Applying isogonal conjugation to the points of a line
yields a conic section through $A,B,C$ (a circumconic of $ABC$),
and every non-degenerate circumconic of $ABC$ can be obtained that way.

*Particularly, applying isogonal conjugation to all points at infinity
(i.e. the line at infinity) yields the circumcircle of $ABC$.

*Extension: The slope of a line (i.e. its point at infinity) determines
the slopes of the principal axes of the circumconic obtained by pointwise
isogonal conjugation of the line.

Given points $D,E$, we can construct their isogonal conjugates $D',E'$
and join them with a line $g = \overline{D'E'}$.
That line is the pointwise isogonal conjugate of the conic section through
$A,B,C,D,E$.
(You could now place a point $F'$ arbitrarily on $g$ and construct its
isogonal conjugate $F$ to obtain another point on the conic.)
Now consider the following cases:

*

*If the line $g$ passes through one of the vertices of the reference triangle
$ABC$, e.g. $A$, then isogonal conjugation runs into a singularity. However,
algebraically, we can still argue that there exists a unique corresponding
circumconic, degenerated to a pair of lines, one of which passes through
that same vertex $A$, and the other matches the opposite side $\overline{BC}$.
Consequently, for a line through two vertices, e.g. $\overline{AB}$, the
corresponding conic consists of the other two extended sides $\overline{BC},
\overline{CA}$.


*If $g$ intersects the circumcircle $U$ of $ABC$ in two distinct real points
$P_1',P_2'$, this means that the corresponding conic intersects
the line at infinity in two distinct points $P_1,P_2$ (at infinity).
A conic with two distinct points at infinity is a hyperbola
(possibly degenerated to a pair of nonparallel lines if case 1 applies),
and those points at infinity identify the slopes of its asymptotes.
Extension: $P_1,P_2$ can be obtained as isogonal conjugates of
$P_1',P_2'$. When breaking this down to more elementary steps,
each $P_i$ will arise as intersection of parallel lines.
Since you only need to know the associated slope,
you are done as soon as you have constructed one of those lines.


*If $g$ touches the circumcircle $U$ of $ABC$ in one real point $P'$,
this means that the corresponding conic has a double point $P$ at infinity.
Such a conic is a parabola
(possibly degenerated to a pair of parallel lines if case 1 applies),
and its point at infinity gives the slope of the parabola's symmetry axis.


*If $g$ does not intersect the circumcircle $U$ of $ABC$,
this means that the corresponding conic contains no point at infinity.
Such a conic is an ellipse.
Extension:
if $g$ is not the line at infinity,
the slopes of the corresponding circumconic's principal axes can be obtained by
determining the points $H_1',H_2'$ on the circumcircle $U$ with tangents
parallel to $g$ and transforming them back to points $H_1,H_2$ at infinity.
The solution to your problem is therefore to construct the isogonal conjugate
$D'$ of $D$ and its tangents to the circumcircle $U$ of $ABC$.
Then, when given another point $E$ of the conic, construct its isogonal
conjugate $E'$ and test whether $\overline{D'E'}$ intersects $U$.
The region for $E'$ where such intersection happens is bounded by the
tangents to $U$ through $D'$ and is tinted green in the figure below.


*

*If $E'$ is inside the green region, the conic is a hyperbola,
possibly degenerated to a pair of nonparallel lines
(if $\overline{D'E'}$ intersects a vertex of $ABC$).

*If $E'$ lies on a bounding tangent, the conic is a parabola,
possibly degenerated to a pair of parallel lines
(if the point of tangency is a vertex of $ABC$).

*Otherwise the conic is an ellipse.

For those bounding tangents to exist, $D'$ must not be inside $U$.
Isogonal conjugation translates this requirement to $A,B,C,D$ being
vertices of a convex 4-gon, as given in the problem statement.
A: One can always construct two parabolas passing through $p_1$, $p_2$, $p_3$, $p_4$ (green and pink in the figure below), each one possibly degenerating into a couple of parallel lines if two opposite sides of quadrilateral $p_1p_2p_3p_4$ are parallel. Point $p_5$ will determine an ellipse if it lies inside either parabola but not in their intersection.
This follows from the fact that five points always determine a conic section, and because the parabola is a limiting case between ellipse and hyperbola: each time $p_5$ crosses the boundary of a parabola, conic section $p_1p_2p_3p_4p_5$ switches from ellipse to hyperbola (or viceversa).

