How is the number of points in the convex hull of five random points distributed?

This is about another result that follows from the results on Sylvester's four-point problem and its generalizations; it's perhaps slightly less obvious than the other one I posted.

Given a probability distribution in the plane, if we know the probability $p_5$ for five points to form a convex pentagon and the probability $p_4$ for four points to form a convex quadrilateral, how can we determine the distribution of the number of points in the convex hull of five points (where all the points are independently drawn from the given distribution)?

Denote the probability for the convex hull of the five points to consist of $k$ points by $x_k$. The convex hull has five points if and only if the five points form a convex pentagon, so $x_5=p_5$.

Now let's determine the expected number of subsets of four of the five points that form a convex quadrilateral in two different ways. There are $5$ such subsets, and each has probability $p_4$ to form a convex quadrilateral, so the expected number is $5p_4$. On the other hand, if the convex hull has $5$ points, all $5$ subsets form a convex quadrilateral; if it has $4$ points, the convex hull itself and two of the other four quadrilaterals are convex, for a total of $3$, and if the convex hull has $3$ points, exactly one of the five quadrilaterals is convex (the one not including the hull vertex that the line joining the two inner points separates from the other two hull vertices). Thus we have

$$5p_4=5x_5+3x_4+x_3\;.$$

Together with $x_5=p_5$ and $x_3+x_4+x_5=1$, that makes three linear equations for the three unknowns. The solution is

\begin{align} x_3&=\frac32-\frac52p_4+p_5\;,\\ x_4&=-\frac12+\frac52p_4-2p_5\;,\\ x_5&=\vphantom{\frac12}p_5\;. \end{align}

MathWorld gives $p_4$ and $p_5$ for points uniformly selected in a triangle and a parallelogram; here are the corresponding distributions:

$$\begin{array}{c|c|c|c} \text{shape}&p_4&p_5&x_3&x_4&x_5\\\hline \text{triangle}&\frac23&\frac{11}{36}&\frac5{36}&\frac59&\frac{11}{36}\\\hline \text{parallelogram}&\frac{25}{36}&\frac{49}{144}&\frac5{48}&\frac59&\frac{49}{144} \end{array}$$

The probability $x_4$ that the convex hull consists of four of the five points is the same in both cases; however, this probability is different for an ellipse. Here's code to check these results and estimate the values for an ellipse.

• Thanks a lot for this really nice question and answer! I am surprised that $x_4$ is the same for both triangle and parallelogram. Do you have an idea If we could prove it directly? Decomposing the parallelogram into $2$ triangles maybe? For the ellipse this is not the same value but did you try your simulations with quadrangle which are not parallelogram? – Gilles Bonnet May 9 '14 at 18:17
• @GillesBonnet: Sorry, I missed your question at the time. I added the quadrangle simulation, and as you perhaps expected, the quadrangle has the same $x_4$ as the triangle and the parallelogram (but different $x_3$ and $x_5$ from either -- for the part of the unit square with $2x\gt y$ the values for $x_3$ and $x_5$ lie between those for the triangle and the parallelogram). – joriki Apr 7 '16 at 17:09
• @GillesBonnet: The event for $x_4$ consists of the $5$ mutually exclusive events of one of the five points being inside a convex quadrangle formed by the other four, so each of these events has probability $\frac19$, which makes it seems even more likely that there should be a more direct proof for this. By the way, I did the simulation for a pentagon, and the $x_4$ value was different. – joriki Apr 7 '16 at 19:55