If $4$ vertices of a regular octagon are selected at random, then the probability that the quadrilateral formed by them is a rectangle is:
(A) $1\over8$
(B) $2\over21$
(C) $1\over32$
(D) $1\over35$
I went about it this way:
We have two types of rectangles, i.e. the blue and green.
If we count the number of rectangles manually, we get 6 rectangles in total.
Counting the number of ways to choose $4$ of $8$ vertices, we get $^8C_4=70$.
So the answer should be $\frac {6}{70}=\frac {3}{35}$. But this is not in the options at all. The answer is (D).
Where is my mistake?
Okay, so I asked my teacher and he solved it like this:
To calculate $n(S)$, where $S$ is sample space, first we select any one vertice, say $A$ in $^8C_1$ ways. Then, for the remaining $3$ vertices of the quadrilateral, there are $^7C_3$ ways. We divide this by $4$ since $A$ can be any of the four vertices of the chosen quadrilateral. $$n(S) = ^8C_1 \times ^7C_3 \times \frac 1 4 = 70$$
Now we count the number of quadrilaterals using $A$ as a vertice. To avoid repetition, we only go counting counterclockwise (for example, we will count $ABEF$ but not $AHED$.
Now there are only $2$ quadrilaterals using $A$. So $n(E)=2$, where $E$ is the event of a rectangle.
Then $$P(E) = \frac {n(E)}{n(S)} = \frac 2 {70} = \frac 1 {35}$$
Which of these approaches is correct? And what is the error in the incorrect one?