If $4$ vertices of a regular octagon are selected at random, then the probability that the quadrilateral formed by them is a rectangle is:

(A) $1\over8$

(B) $2\over21$

(C) $1\over32$

(D) $1\over35$

I went about it this way:

We have two types of rectangles, i.e. the blue and green.

One of each

If we count the number of rectangles manually, we get 6 rectangles in total.

All rectangles

Counting the number of ways to choose $4$ of $8$ vertices, we get $^8C_4=70$.

So the answer should be $\frac {6}{70}=\frac {3}{35}$. But this is not in the options at all. The answer is (D).

Where is my mistake?

Okay, so I asked my teacher and he solved it like this:

To calculate $n(S)$, where $S$ is sample space, first we select any one vertice, say $A$ in $^8C_1$ ways. Then, for the remaining $3$ vertices of the quadrilateral, there are $^7C_3$ ways. We divide this by $4$ since $A$ can be any of the four vertices of the chosen quadrilateral. $$n(S) = ^8C_1 \times ^7C_3 \times \frac 1 4 = 70$$

Now we count the number of quadrilaterals using $A$ as a vertice. To avoid repetition, we only go counting counterclockwise (for example, we will count $ABEF$ but not $AHED$.

One of each

Now there are only $2$ quadrilaterals using $A$. So $n(E)=2$, where $E$ is the event of a rectangle.

Then $$P(E) = \frac {n(E)}{n(S)} = \frac 2 {70} = \frac 1 {35}$$

Which of these approaches is correct? And what is the error in the incorrect one?

  • $\begingroup$ First thing you are counting $4$ rectangles and $2$ squares. That could be one of the differences. $\endgroup$
    – Math Lover
    Jun 3, 2021 at 12:35
  • 1
    $\begingroup$ @MathLover Even if I counted only the non-square rectangles, the answer would be $\frac 2 {35}$ $\endgroup$
    – Righter
    Jun 3, 2021 at 12:44
  • 1
    $\begingroup$ Yes you are right on that. I noticed that too. I do not see how the answer could be $\frac{1}{35}$. $\endgroup$
    – Math Lover
    Jun 3, 2021 at 12:45
  • $\begingroup$ It depends on what exactly is meant by quadrilateral, since some definition might allow self-intersection quadrilaterals. The quadrilateral with vertices F,E,B,A could have edges FE, EB, BA and AF which would make it a rectangle, but the quadrilateral with edges FB, BA, AE, EF would not be a rectangle. $\endgroup$ Jun 3, 2021 at 12:58
  • $\begingroup$ With this interpretation we would divide the number by three, since for any 4 edges there would only be $8$ out of the $4!=24$ arrangements which would be a rectangle. $\endgroup$ Jun 3, 2021 at 13:07

1 Answer 1


Yours is right.

A big problem with your teacher's argument is that they forgot about the quadrilateral ADEH. Even so, it is not quite right, since I am not even sure what they are counting.

So, their n(S) is precisely 70, since they are actually just choosing 4 vertices out of 8, and then giving them a well-defined order. If they want to count it on a per-vertex basis, they would multiply that by $4$ (to allow for rotations), and say that each vertex gives rise to $3$ quadrilaterals, giving a final answer of $\frac{3*8}{70*4}$ which matches your answer.

I think the question askers got the 1/35 by using the following argument: you can pick any vertex as the first vertex, and then you can pick one of 6 of 7, and then 1/6 and then 1/5 vertices, giving you 1/35. This is wrong, and the fundamental difference is that not every permutation of the selection of the vertices gives rise to a rectangle, so selecting them one by one is indeed different than selecting 4 from a set and then checking. It is quite subtle.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .