# Probability of a random polygon that does not contain the center of a regular polygon

Problem:

Let $m$ be an even number such that $m \geq 6$, and let $n$ be a natural number such that $3 \leq n \leq \cfrac{m}{2}$. Number each vertex of a regular $m$-sided polygon clockwise as $1,2,\cdots,m$. Randomly choose $n$ vertices among the $m$ vertices and construct an $n$-sided polygon. We assume that two $n$-sided polygons that do not share at least one vertex are different from each other. Congruent polygons anchored on different vertices (rotated or reflected) are considered different. Let $P_{m,n}$ be the probability that the center of the $m$-sided polygon is not located inside nor on an edge of the $n$-sided polygon. Then, find the probability $P_{m,n}$.

What I tried:

First, I calculated $P_{m,n}$ for $(m,n)=(8,3),(8,4),(10,3),(10,4)$:
$P_{8,3} = \cfrac{3 \times 8}{_8C_3} \approx 0.4286$
$P_{8,4} = \cfrac{1 \times 8}{_8C_4} \approx 0.1143$
$P_{10,3} = \cfrac{6 \times 10}{_{10}C_3} = 0.5$
$P_{10,4} = \cfrac{4 \times 10}{_{10}C_4} \approx 0.1905$
For example, when $(m,n)=(8,3)$, we can freely choose vertex 1 by symmetry. And we choose two other vertices $v_1, v_2 (v_1 < v_2)$ among the half side of the regular $8$-sided polygon so that we can simply multiply the combinations by 8. For $v_1 = 2$ we must have $v_2 = 3,4$. For $v_2 = 3$ we must have $v_2 = 4$. Therefore, $P_{8,3} = \cfrac{3 \times 8}{_8C_3}$.

For $n=3$, the possible combination of the triangle is:
$m=8 \Rightarrow (1 + 2) \times 8 = 24$
$m=10 \Rightarrow (1 + 2 + 3) \times 10 = 60$
Thus, $$P_{m,3} = \cfrac{m \sum^{(\frac{m}{2}-3)+1}_{k=1}{k}}{_mC_3} = \cfrac{\cfrac{m}{2}(\cfrac{m^2}{4}-\cfrac{3m}{2}+2)}{\cfrac{m(m-1)(m-2)}{3 \cdot 2 \cdot 1}} = \cfrac{3}{4}\cdot\cfrac{m^2-6m+8}{m^2-3m+2}$$

However, I don't know how to generalize it and find $P_{m,m}$. Could someone help me with this problem?

• Could you clarify: "We assume that two $n$-sided polygons that do not share at least one vertex are different from each other." Does that mean the following? "As long as there exist any vertex of one polygon that doesn't also belong to the other, then we say the two polygons are different". To double check, this also means that congruent polygons anchored on different vertices (rotated or reflected) are considered different, right? – Lee David Chung Lin Apr 14 '18 at 1:20
• Yes, that's exactly what I wanted to say. – ptr-yudai Apr 15 '18 at 1:53

You are then free to choose any $n-1$ of the $\frac{m}2-1$ points anticlockwise from the first point.