# Why is observing 100 heads for a fair coin flips surprising?

Assume that we have a fair coin. We flip it 100 times. The outcome is all heads.

Why is it surprising?

Doesn't all outcomes have the same probability? Any particular outcome, including irregular ones, would have the same very small probability.

Why is it that observing an irregular outcome is less surprising to us than a regular one?

• The probability of getting heads, given that the previous one was heads, should be 1/2. If you have a "genuine" outcome of more randomized looking tosses, then conditioning on the previous outcomes, it behaves independently of the previous ones. Mar 28 at 18:02
• The probability of that happening is $\frac{1}{2^{100}}$. If you pour a gallon of milk on the floor there is a chance that it will form the shape of the jug. If that happened I bet most of us would evacuate ourselves. Mar 28 at 18:03
• @Kaveh if you are thinking about a real-world situation it might be interesting to look at the "ludic fallacy", see en.m.wikipedia.org/wiki/Ludic_fallacy Mar 28 at 18:06
• I'd be more surprised by HTHTHT...HT. All heads is easily explained: the tosser slipped in a two-headed coin. For heads and tails to alternate like that, either he must be able to control the outcome, or he's somehow alternating between a two-headed and a two-tailed coin. Mar 28 at 18:19
• Mar 28 at 20:37

You are right. It is not that surprising. Each specific pattern has probability $$\frac{1}{2^{100}}$$. The probability to obtain $$100$$ heads is the same as obtaining any other particular outcome.

But usually we do not ask for a specific pattern but ask for the probability to obtain e.g. $$k$$ heads and this makes the difference.

• There is just $$\color{blue}{\binom{100}{100}=1}$$ pattern to obtain $$100$$ heads out of $$2^{100} \sim 1.3\cdot 10^{30}$$.

• But we have $$\color{blue}{\binom{100}{50}\sim 1.0\cdot 10^{29}}$$ patterns which contain $$50$$ heads and $$50$$ tails.

If the sequences that you consider "irregular" constitute the vast majority of all possible sequences, then the probability that the sequence you get is irregular is very high, and it's surprising if you get one of the relatively few "regular" sequences.

One could make the concept of "irregular" precise using something like Kolmogorov complexity. Most bit strings of a given length have no description shorter than the string itself; the ones that admit short descriptions are relatively few.

• yes, if I was comparing regular vs. irregular events, you would be right. I am taking about single outcomes. A particular random looking sequence does not surprise us, even though it has exactly the same probability as the all heads one. Mar 28 at 20:46
• @Kaveh and I'd argue that if you really find the "all heads" outcome surprising, it's because you are measuring regularity. Our brains are naturally inclined to notice patterns and come up with models for the phenomena we observe, and when we notice that the sequence has a simple description, it makes us suspect that the mechanism underlying the coin tosses is less random than we thought - that's the surprise.
– Karl
Mar 28 at 20:56
• I.e. the "surprise" is the tension between (1) our natural temptation to accept the low-entropy, high-predictive-power "always heads" model as the true mechanism, and (2) the (presumably strong) conviction that the true model is the higher-entropy, less-predictive "fair coin" model that was originally assumed.
– Karl
Mar 28 at 21:05
• Possibly related: minimum description length
– Karl
Mar 28 at 21:15

Assume a fair coin. The idea of "surprising" means it's against our "expectations". The distinction is what is our "expectation"?

• If it were a specific exact sequence of heads and tails, then the all heads sequence is just as likely as any other specific sequence, $$2^{-100}$$. This is a very rare thing to "expect".

• More likely our expectation is specified as something like "roughly an even number of heads and tails", which is not one specific sequence of coin tosses but the event of seeing one of many, many sequences. We can calculate with the binomial theorem or approximating a Normal distribution that the probability of "seeing roughly and even number of heads and tails" is very high. For example, the probability of seeing 40-60 heads in our 100 tosses is $$2^{-100} \sum_{i=40}^{60} \binom{100}{i} \approx (1.22 \times 10^{30}) / (1.26 \times 10^{30}) \approx 96.4\%$$

You're asking for a specific sequence of coin tosses of length 100. Sure, any such sequence is as rare as all others, but there are $$2^{100}$$ of them (all heads, all tails, HTHTHTHTHTH...HTHT, HHHTTTHHHTTT...TTH, etc). So picking one specific sequence means getting it right FOR EVERY SINGLE THROW. This is almost impossibly improbable ($$P(\omega) = \frac{1}{2^{100}}$$). If you were able to do that in a casino, starting from a "double-or-nothing" bet of 1\$ you'd own all the money, debt contracts, resources, and companies in the world long before the end of the coin tosses.

Take two independent events $$A$$ and $$B$$. Then the probability of both happening, which we'll write as $$P(A\cap B)$$, is given by $$P(A)\times P(B)$$. In general, given $$n$$ independent events $$E_n$$ with probability $$p_n$$, the total probability of all of them happening is

$$P(\bigcap_{i=1}^nE_n) = \prod_{i=1}^np_n$$

In this case we have 100 independent events, the coin flips into a head, all happening with propability $$p=\frac{1}{2}$$, meaning we need to multiply $$\frac{1}{2}$$ by itself one hundred times, namely $$(\frac{1}{2})^{100} = \frac{1}{2^{100}}$$, roughly $$7.9 \times 10^{-31}$$.

Is such an outcome surprising? Well, it would be.

According to https://en.wikipedia.org/wiki/Mega_Millions the probability of winning is about $$1$$ in $$300$$ million. Three hundred million is approximately $$2^{28}$$ and $$2^{100}=2^{72}*2^{28}$$.

$$2^{72}\approx 4.72\times 10^{21}$$

Would you be surprised if you won the Megamillions that many times in a row?

All $$2^{100}$$ sequences are equally probable.

The sequences "all heads" and "all tails" are the most easily specified and most easily noticed sequences.

Imagine you have the coins numbered from 1 to 100 to confirm that the sequence is "all heads" all you need to do it to count zero tails.

The sequence "all coins whose number is a multiple of 5 are heads, the others tails" is equally probable but much more difficult to confirm.

To confirm, you would need to...

• read the numbers on all the coins that show heads to confirm that these are all multiples of 5.

A Rosencrantz and Guildenstern type of game (I get the heads, you get the tails) is fairly typical of the most common coin tossing situations insofar as the thing you are most interested in is the total numbers of heads.

The specifications "all heads" and "all tails" are the only ones for which specifying the number of heads is sufficient to determine the entire sequence.

Maybe the reason is

P(the coin is fair | a sequence with low Kolmogorov complexity) $$\ll$$ P(the coin is fair | a Martin-Lof random sequence)

Although

P(a sequence with low Kolmogorov complexity | the coin is fair) = P(a Martin-Lof random sequence | the coin is fair) = $$\frac{1}{2^{100}}$$

yet if we take the Solomonoff prior, we would use the algorithmic probability to estimate

$$P(x)\approx 2^{-K(x)}$$

then

P(a sequence with low Kolmogorov complexity) $$\gg$$ P(a Martin-Lof random sequence)

therefore

P(the coin is fair | a sequence with low Kolmogorov complexity) $$\ll$$ P(the coin is fair | a Martin-Lof random sequence)

You can't flip it $$100$$ times and get the same outcome as head, but you can flip it $$1$$ to get outcome as head

You can't flip it $$50$$ times and get the same outcome as head, but you can flip it twice to get same outcome as head

You can't flip it $$25$$ times and get the same outcome as head, but it can happen as chance to get same outcome as head in $$4$$ flip

It's unlikely that chance can happen or you get same result for $$8$$ flips, because the more the flips the more, the likely the outcome changes

The possibility you get a head or tail is $$\frac{1}{2}$$ because there are only two choice... This means if I flip it $$n$$ times, I will likely have equal number of both heads and tails

If I flip the coin $$n$$ times, and I section the outcomes into $$\frac{n}{p}$$ in $$p$$ times, the outcome wouldn't have same possiblity, In fact every new flip behaves independent of the previous outcome

It is surprising because we can recognise a pattern, making the observed sequence remarkable.

Certainly, any other sequence has the same tiny probability, but we are not surprised by most of them, because we cannot recognise a pattern, and so do not find them remarkable. But we should also be surprised – if we noticed them – by other remarkable sequences of a hundred tosses, such as:

• strictly alternating heads and tails
• a palindromic sequence, i.e. one which is the same backwards
• any pattern repeated all through, ee.g. the second fifty the same as the first or HHTHTT 16⅔ times