Number of vertices of a random convex polygon Take $n>2$ random points, chosen independently with uniform probability on $[0,1]\times[0,1]$. What is the probability $P(n,k)$ that the convex hull of these points is a polygon with exactly $2<k\leq n$ vertices?
It seems that $P(3,3)=1$, since after choosing two points, the probability that the third point will be on the line between them is $0$. 
For $P(4,4)$, assume 3 points have already been chosen, then their convex hull is a triangle which is the intersection of 3 half spaces. The fourth point will create a quadrilateral if and only if it lies in the intersection of exactly two of the above half spaces. But how can we calculate this area? And what if $n>4$? Then things get really tricky!
 A: This is a very broad question. $P(n,n)$ and $P(n,3)$ are known, and this allows $P(5,4)$ to be deduced. I'm not aware that any other values are known.
For $k=n$, the answer is given in the dissertation Probability that $n$ random points are in convex position (Pavel Valtr, 1994):
$$
P(n,n)=\left(\frac{\binom{2n-2}{n-1}}{n!}\right)^2\;.
$$
For $k=3$, the event that the convex hull consists of three points is the union of the disjoint events that one of the $\binom n3$ triangles formed by $3$ of the $n$ points contains the remaining $n-3$ points. The probability for each of these events is the $(n-3)$-rd power of the probability that a point lies in the triangle formed by three points, which is the $(n-3)$-rd moment of the area of the triangle. These moments are given at MathWorld, and multiplying them by $\binom n3$ yields
$$
P(n,3)=2^{5-n}\frac{(n-1)H_{n-2}+1}{n(n-1)^2}\;,
$$
where $H_n$ is the $n$-th harmonic number.
From $P(5,3)+P(5,4)+P(5,5)=1$ we can also deduce $P(5,4)=\frac59$.
Here are the values for $n\le5$:
\begin{array}{c|cc}
n\setminus k&3&4&5\\\hline
3&1\\
4&\frac{11}{36}&\frac{25}{36}\\
5&\frac5{48}&\frac59&\frac{49}{144}\\
\end{array}
You can read more about them at Expected size of subset forming convex polygon. and at How is the number of points in the convex hull of five random points distributed?, where the distribution $P(5,k)$ is derived in a different way.
