# Probability of making rectangles by choosing $4$ points of regular octagon.

If $$4$$ vertices of a regular octagon are selected at random, then the probability that the quadrilateral formed by them is a rectangle is:

(A) $$1\over8$$

(B) $$2\over21$$

(C) $$1\over32$$

(D) $$1\over35$$

I went about it this way:

We have two types of rectangles, i.e. the blue and green.

If we count the number of rectangles manually, we get 6 rectangles in total.

Counting the number of ways to choose $$4$$ of $$8$$ vertices, we get $$^8C_4=70$$.

So the answer should be $$\frac {6}{70}=\frac {3}{35}$$. But this is not in the options at all. The answer is (D).

Where is my mistake?

Okay, so I asked my teacher and he solved it like this:

To calculate $$n(S)$$, where $$S$$ is sample space, first we select any one vertice, say $$A$$ in $$^8C_1$$ ways. Then, for the remaining $$3$$ vertices of the quadrilateral, there are $$^7C_3$$ ways. We divide this by $$4$$ since $$A$$ can be any of the four vertices of the chosen quadrilateral. $$n(S) = ^8C_1 \times ^7C_3 \times \frac 1 4 = 70$$

Now we count the number of quadrilaterals using $$A$$ as a vertice. To avoid repetition, we only go counting counterclockwise (for example, we will count $$ABEF$$ but not $$AHED$$.

Now there are only $$2$$ quadrilaterals using $$A$$. So $$n(E)=2$$, where $$E$$ is the event of a rectangle.

Then $$P(E) = \frac {n(E)}{n(S)} = \frac 2 {70} = \frac 1 {35}$$

Which of these approaches is correct? And what is the error in the incorrect one?

• First thing you are counting $4$ rectangles and $2$ squares. That could be one of the differences. Commented Jun 3, 2021 at 12:35
• @MathLover Even if I counted only the non-square rectangles, the answer would be $\frac 2 {35}$ Commented Jun 3, 2021 at 12:44
• Yes you are right on that. I noticed that too. I do not see how the answer could be $\frac{1}{35}$. Commented Jun 3, 2021 at 12:45
• It depends on what exactly is meant by quadrilateral, since some definition might allow self-intersection quadrilaterals. The quadrilateral with vertices F,E,B,A could have edges FE, EB, BA and AF which would make it a rectangle, but the quadrilateral with edges FB, BA, AE, EF would not be a rectangle. Commented Jun 3, 2021 at 12:58
• With this interpretation we would divide the number by three, since for any 4 edges there would only be $8$ out of the $4!=24$ arrangements which would be a rectangle. Commented Jun 3, 2021 at 13:07

So, their n(S) is precisely 70, since they are actually just choosing 4 vertices out of 8, and then giving them a well-defined order. If they want to count it on a per-vertex basis, they would multiply that by $$4$$ (to allow for rotations), and say that each vertex gives rise to $$3$$ quadrilaterals, giving a final answer of $$\frac{3*8}{70*4}$$ which matches your answer.