Ellipse inscribed in an irregular quadrilateral I want to obtain the ellipse inscribed in the irregular quadrilateral (no parallel sides) defined by the four points A, B, C, D.

I summarize the ideas given in the comments and answers:

*

*The is not an unique ellipse inside the given quadrilateral.

*For the unit square, there are infinite ellipses inscribed into it, with different eccentricities

*You cannot transform the unit square into a irregular quadrilateral using linear transformations, as those transform only two vectors into other two vectors. In this case we need to transform 4 vectors.

As shown in this figure:

Increasing the eccentricity, decreases the area. So the problem can be reduced to obtain the maximum area ellipse inscribed into the quadrilateral.
 A: *

*There is unique inscribed ellipse of a convex pentagon (dual case for $5$ points defining a conics).  There are one and two degrees of freedom of drawing an inscribed ellipse in a (convex) quadrilateral and triangle respectively.


*By means of skew transformation, we can transform an irregular quarilateral (convex but not parallelogram) into one with one pair of opposite sides are perpendicular.
$$(x',y')=(x+y\cos \omega,y\sin \omega)$$

*

*Taking the vertices as $A(a,0)$, $B(b,0)$, $C(0,c)$ and $D(0,d)$ where $ab>0$, $cd>0$ and $(a-b)(d-c)>0$.


*The two extreme cases are the ellipse degenerates into the diagonals.


*Construct a family of conics touching with the axes with parameter $k$:
$$
\left[
  k\left( \frac{x}{a}+\frac{y}{c} \right)+
  (1-k)\left( \frac{x}{b}+\frac{y}{d} \right)-1
\right]^2=\lambda x y \tag{$\star$} $$

*

*Using a discriminant to check tangency for $\frac{x}{a}+\frac{y}{d}=1$ or $\frac{x}{b}+\frac{y}{c}=1$, we can solve for $\lambda$.

$$\lambda=4k(1-k)
\left(
  \frac{1}{a}-\frac{1}{b}
\right)
\left(
  \frac{1}{d}-\frac{1}{c}
\right)$$

*

*For ellipse,

$$4k(1-k)
\left(
  \frac{1}{a}-\dfrac{1}{b}
\right)
\left(
  \frac{1}{d}-\dfrac{1}{c}
\right)
\left(
  \frac{k}{ac}+\frac{1-k}{bd}
\right)>0 \implies k\in (0,1) $$

*

*The centre of the ellipse lies on the Newton line which is the line joining the mid-points of the diagonals.

$$\text{centre}=\frac{
\left(
  \dfrac{k}{c}+\dfrac{1-k}{d},\dfrac{k}{a}+\dfrac{1-k}{b}
\right)}{2
\left(
  \dfrac{k}{ac}+\dfrac{1-k}{bd}
\right)}$$

*

*See also another post of mine for the case of triangle here.


*An illustration of a tangential quadrilateral.  Note on the circular case at $k=0.6$:

Addendum

*

*To generalize to any kind of convex quadrilateral, we may use the skew axes as the diagonals.  Now taking the vertices as $A(a,0)$, $B(0,b)$, $C(c,0)$ and $D(0,d)$ where $ac<0$ and $bd<0$.


*In tangential coordinates $(X,Y)$, tangent line $\frac{x}{a}+\frac{y}{b}=1$ can be written as $$Xx+Yy+1=0$$


*Hence, the dual conic will pass through a "rectangle" with vectices $(-\frac{1}{a},-\frac{1}{b})$, $(-\frac{1}{c},-\frac{1}{b})$, $(-\frac{1}{c},-\frac{1}{d})$ and $(-\frac{1}{a},-\frac{1}{d})$, that is
$$\lambda (aX+1)(cX+1)+\mu (bY+1)(dY+1)=0$$

*

*Let $(\lambda,\mu) \propto (1-k,k)$, the inscribed ellipse is

$$\det
\begin{pmatrix}
  0 & x & y & 1  \\
  x & \lambda ac & 0 & \frac{\lambda (a+c)}{2} \\
  y & 0 & \mu bd & \frac{\mu (b+d)}{2} \\
  1 &\frac{\lambda (a+c)}{2} & \frac{\mu (b+d)}{2} & \lambda+\mu
\end{pmatrix}=0$$

*

*The centre divides the Newton line, from $(0, \frac{b+d}{2})$ to $(\frac{a+c}{2},0)$ internally with ratio $\lambda:\mu$


*Illustration of dual conics pair:


A: A linear transformation may not exist between your quadrilateral and the unit square. There exists a unique linear transformation that sends two given vectors to two other vectors. so the other two position vectors of the quadrilateral may not transform accordingly. An interesting question, note that the ellipse is simply a stretching of the circle by some factor.
A: As explained in this answer, once you choose one of the tangency points the other three can be uniquely constructed from the properties of the perspective transformation carrying the quadrilateral to a square.
If you want, in particular, the ellipse corresponding to the circle inscribed into the square, then the construction is very simple (see figure below): join the intersection point $O$ of the diagonals with the concurrence points $E$, $F$ of the opposite sides: lines $EO$ and $FO$ intersect the sides of quadrilateral $ABCD$ at tangency points $PQRS$.
One can then find the centre $M$ of the ellipse as the intersection of lines $DK$, $CL$, where $K$, $L$ are the midpoints of $QR$, $PQ$. If you need to find the equation of the ellipse, you can construct a fifth point $Q'$ as the reflection of $Q$ about $M$. Otherwise, you can use geometric techniques to find axes and foci.

A: The general equation of an ellipse can be written as $$x^2+Axy+By^2+Cx+Dy+E=0$$
(assuming $A^2<4B$)
So there are five unknown coefficients.
Each tangent line is of the form $y=mx+c$. Solving each simultaneously with the conic gives a quadratic equation in $x$ which must have double roots.
So you will have only four equations in the five unknowns $A, B, C,D$ and $E$, and this would suggest infinitely many possible ellipses can be inscribed in the quadrilateral.
So you wouldn't be able to find a unique ellipse without applying some additional constraint.
