Inscribing an ellipse in an irregular convex pentagon Using the methods of projective geometry, identify the unique ellipse that is inscribed in a given convex pentagon.
Suppose the vertices of the pentagon are: $(1, 0), (4, 2), (3, 6), (-1, 5), (-1, 1)$.  Find the equation of the unique inscribed ellipse that is tangent to all five sides of this convex pentagon.

 A: It's easier to find the dual conic first.
Rewrite every tangents in the form:
$$X_i x+Y_i y+1=0$$
Then $(X_i,Y_i)$ are the five points defining the dual conic:
$$
\begin{align}
  0 &= \det
  \begin{pmatrix}
    X^2 & XY & Y^2 & X & Y & 1 \\
    X_1^2 & X_1 Y_1 & Y_1^2 & X_1 & Y_1 & 1 \\
    \vdots & & & & & \vdots \\
    X_5^2 & X_5 Y_5 & Y_5^2 & X_5 & Y_5 & 1 \\
  \end{pmatrix} \\ \\
  &= AX^2+2HXY+BY^2+2GX+2FY+C \\
\end{align}$$
The required ellipse is
$$
\begin{align}
  0 &= -\det
  \begin{pmatrix}
    0 & x & y & 1 \\
    x & A & H & G \\
    y & H & B & F \\
    1 & G & F & C \\
  \end{pmatrix} \\ \\
  &= ax^2+2hxy+by^2+2gx+2fy+c
\end{align}$$
Note that $a$ is the co-factor of entry $A$ of the $3\times 3$ block matrix, etc.
For the vertical tangent, try to let $Y_i=N$ and only the terms with highest order in $N$ survive.
See also the case of quadrilateral here.
A: *

*Construct points $B', C', D', E'$ such that pentagon $(P'):=A B'C'D'E'$ is regular, convex and direct. This is easily done by rotating vector $\vec{OA}$ by successive angle $k2 \pi/5$.


*Determine the inscribed circle $(C)$ of pentagon $(P')$.


*Find the (unique) projective transform $(T)$ mapping quadrilateral $BCDE$ onto quadrilateral $B'C'D'E'$ (see Appendix below).


*The desired ellipse is the image of circle $(C)$ by transform $(T)$.
Appendix : how to find the unique projective transform mapping  $B',C',D',E'$ onto $B,C,D,E$ resp. ?
We are looking for coefficients $a,b,c,d,e,f,g,h,i$ such that :
$$x=\frac{ax'+by'+c}{gx'+hy'+i}, \ \ \ y=\frac{dx'+ey'+f}{gx'+hy'+i} \tag{1}$$
(please note the common denominator in (1); if we take $e=f=0$, we are in the particular case of an affine transform).
Writing the two constraints for each of the four points generates a homogeneous system of $2 \times 4 = 8$ equations in the $9$ unknowns $a,b,c,d,e,f,g,h,i$ : this system has a unique non-zero solution up to a multiplicative constant, (this constant vanishes when we apply formulas (1)).
