It's intuitive that a concave quadrilateral cannot be circumscribed by an ellipse nor by a parabola.

I can prove it by analytic geometry:

Without loss of generality, let $L_1\equiv m_1x -y +r_1=0$, $L_2\equiv m_2x -y +r_2=0$, $L_3\equiv m_3x -y +r_3=0$, $L_4\equiv m_4x -y +r_4=0$ be the equations of lines $AB$, $BC$, $CD$, $DA$ of a concave quadrilateral ABCD.

Since ABCD is a concave quadrilateral, by virtue of the theorem stated and proved in The concave quadrilateral and the slopes of its sides, we have


Besides that, all the conics which circumscribe the concave quadrilateral ABCD can be given by the equation $kL_1L_3+L_2L_4=0$ (except the degenerate conic consisting of the pair of concurrent lines $AB$ and $CD$). This degenerate conic is a degenerate hyperbole because a concave quadrilateral cannot have parallel opposite sides.

Therefore all the conics circumscribing the quadrilateral ABCD (except the mentioned degenerate hyperbole) are given by the equation $$k(m_1x -y +r_1)(m_3x -y +r_3)+(m_2x -y +r_2)(m_4x -y +r_4)=0,$$ $$(m_1m_3k +m_2m_4)x^2-((m_1+m_3)k+(m_2+m_4))xy+(k+1)y^2+...=0$$

The type of this circumscribing conic is given by

$$\delta=(km_1m_3+m_2m_4)(k+1)-\frac 14(k(m_1+m_3)+(m_2+m_4))^2,$$

This circumscribing conic is an ellipse if $\delta>0$, an hyperbole if $\delta<0$, a parabola if $\delta=0$.

Developing $\delta$ we get

$$4\delta= 4m_1m_3k^2+4m_2m_4k+4m_1m_3k+4m_2m_4-((m_1+m_3)^2k^2+2(m_1+m_3)(m_2+m_4)k+(m_2+m_4)^2),$$ $$4\delta=-(m_1-m_3)^2k^2+2[2m_2m_4+2m_1m_3-(m_1+m_3)(m_2+m_4)]k-(m_2-m_4)^2,$$


Now let $\psi$ be the discriminant of this second degree equation in k: $$\psi=4[(m_1-m_3)(m_2-m_4)+2(m_1-m_4)(m_3-m_2)]^2-4(m_1-m_3)^2(m_2-m_4)^2,$$ $$\frac 14 \psi=[(m_1-m_3)(m_2-m_4)+2(m_1-m_4)(m_3-m_2)]^2-(m_1-m_3)^2(m_2-m_4)^2,$$ $$\frac 14 \psi=4(m_1-m_3)(m_2-m_4)(m_1-m_4)(m_3-m_2)+4(m_1-m_4)^2(m_3-m_2)^2,$$ $$\frac 1{16} \psi=(m_1-m_3)(m_2-m_4)(m_1-m_4)(m_3-m_2)+(m_1-m_4)^2(m_3-m_2)^2,$$ $$\frac 1{16} \psi=(m_1-m_4)(m_3-m_2)[(m_1-m_3)(m_2-m_4)+(m_1-m_4)(m_3-m_2)],$$ $$\frac 1{16} \psi=(m_1-m_4)(m_3-m_2)(m_1-m_2)(m_3-m_4),$$

$$\frac 1{16} \psi=(m_1-m_2)(m_2-m_3)(m_3-m_4)(m_4-m_1)$$

Hence this discriminant $\psi$ is negative and since $-(m_1-m_3)^2<0$, $\delta$ is always negative. Consequently a concave quadrilateral can only be circumscribed by hyperboles, therefore it cannot be circumscribed by an ellipse nor by a parabola,


Is there another formal proof of it?


I have no idea about the parabola, I'll only cover the case of ellipse.

Since "concaveness" is preserved under linear transform, we can use a linear transform to map the ellipse to the unit circle centered at origin. The question becomes "can we inscribe a concave quadrilateral into that unit circle?".

Let $ABCD$ be a concave simple quadrilateral. Since it is concave, one of its vertices, say $D$, lies in the interior of the triangle $ABC$. This means we can find $\alpha, \beta, \gamma > 0$ such that $$D = \alpha A + \beta B + \gamma C\quad\text{ and }\quad\alpha+\beta+\gamma + 1$$

Let $\lambda = \alpha+\beta$ and $\mu = \frac{\alpha}{\lambda}$, we have $\lambda,\mu \in (0,1)$ and $(\alpha, \beta, \gamma) = (\lambda\mu,\lambda(1-\mu),1-\lambda)$

Let $\varphi(P)$ be the squared distance of point $P$ from origin. When $A,B,C$ lies on the unit circle, we have $\varphi(A) = \varphi(B) = \varphi(C) = 1$. In order to inscribe $ABCD$ into the unit circle, we also need $\varphi(D) = 1$.

As a function of $P$, $\varphi(P)$ is strictly convex over the Euclidean plane. This implies

$$\begin{align} \varphi(D) &= \varphi(\alpha A + \beta B + \gamma C) = \varphi(\lambda(\mu A + (1-\mu)B) + (1-\lambda) C)\\ &< \lambda \varphi(\mu A + (1-\mu)B) + (1-\lambda) \varphi(C)\\ &< \lambda( \mu \varphi(A) + (1-\mu)\varphi(B)) + (1-\lambda) \varphi(C)\\ &= \lambda( \mu + (1-\mu) ) + 1-\lambda\\ &= 1 \end{align} $$ As a result, $D$ falls inside the interior of unit disk and cannot lie on the unit circle.


I find an argument that works for both parabola and ellipse. The key is the concept extreme point of a convex set. There are several easy to verify facts.

  1. If $D$ lies in the interior of triangle $ABC$, then $D$ is not an extreme point for the triangle (which is a convex set).
  2. If $P \in X \subset Y$ for convex sets $X$ and $Y$, then $P$ is an extreme point for $Y$ implies $P$ is an extreme point for $X$.

  3. The region bounded by an ellipse is convex and the ellipse is the set of extreme points for this region.

Let's say we have an ellipse $\mathcal{E}$ circumscribe a concave quadrilateral $ABCD$. Since $ABCD$ is concave, one of its vertices, say $D$ lies in the interior of triangle $ABC$.

  • By $(1)$, $D$ is not an extreme point for triangle $ABC$.
  • By $(2)$, $D$ is not an extreme point for the region bounded by $\mathcal{E}$ (since this region contains triangle $ABC$).
  • By $(3)$, $D$ doesn't lie on $\mathcal{E}$.

This contradict with the assumption that $\mathcal{E}$ circumscribe $ABCD$.

For parabola, the argument is similar. A parabola divides the plane into two components. One component of it is convex while the other concave. It is easy to see if a parabola $\mathcal{P}$ circumscribe an quadrilateral $ABCD$, then the interior of the edges $AB$, $BC$, $CD$, $DA$ and body of quadrilateral lies inside the convex component.

We can replace $(3)$ by an alternate fact.

  1. The parabola is the set of extreme points for the corresponding convex component.

Essentially same argument like before tell us if $ABCD$ is concave, then the point $D$ (the one lies inside the triangle formed by other points) cannot lies on $\mathcal{P}$. A contradiction again!

  • $\begingroup$ Jensen's inequality...interesting. $\endgroup$ – MrDudulex Oct 21 '18 at 22:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.