# probabilistically what can we say about the next throw of a coin after n throws

this may sound easy or hard or whatever but i cant seem to find anything after searching around for a similar question/answer

The question is this:

What can we say (probabilistically) about the next throw of a (fair) coin after having observed $n$ (the exact value of $n$ is left unspecified, use as you wish) previous throws??

There are 2 possible answers:

1. Since the coin is fair (although irrelevant to the main question) the probability of a next throw (lets say tails) equals $\frac{1}{2}$

2. Since by entropy (or ergodic) considerations a sequence of (fair) coin throws will have to manifest (in the sequence) the relative probabilities of the various outomes as expected (meaning $P(HEADS)=\frac{1}{2}$, $P(TAILS)=\frac{1}{2}$). So in the course of the sequence (the value of $n$ here might be relevant) there should be some point where the next throw should have different probabilities for head and tails (considering the above rationale)

A possible objection is that entropy and ergodicity (related) may be manifested in the limit towards infinity (so for a practical $n$, it could have no observational difference)

However the opposite argument can also be made, from 2 angles

a. it seems awkward (it can be put in more mathematical form) that abruptly at some point $n$ the probabilities will "change" (quotes intended, since thet are not the original probabilities of heads and tails)

b. "typical sequences" (in the context of entropy) have a structure where the outcomes are interspersed in the sequence more densely (for clarifying this point consider the digits of various prime numbers and their distribution)

Last another point of the question is how much should one observe a sequnence before deciding (with a probability threshold) that a coin is NOT fair?

Thanks

UPDATE:

1. sorry for delaying to comment on the given answers (so far), it seems no notification was sent

2. regarding first 2 answers, i tried to enhance the meaning of the question in the comments and explain why they dont point to the main question (the wikipedia article posted was close though)

UPDATE 2:

a related question if that helps to enhance understanding is trying to figure out how random a (black box) random genarator is, this is very similar instead relying on basic probability and entropy (and their "interplay" ) although it is phrased as a gambler's fallacy

UPDATE 3:

saying one option is correct but without trying to explain why the other is not correct or not applicable, is not what the question is about

the question actually "doubts" this, either one is correct and the other is in some way not correct or not applicable or there is some synthesis of the two or maybe sth else?

-
I recommend two readings: (1) Laplace's Rule of Succession at en.m.wikipedia.org/wiki/Rule_of_succession and (2) statweb.stanford.edu/~susan/papers/headswithJ.pdf. The latter deals with the physics of coins and how they are slightly more likely to land the same way twice just because of the normal coin flipping motion. – Jason Zimba Apr 1 '14 at 0:24
@JasonZimba, thanks the wikipedia article is good on this, it seems i missed that, i could accept it as an answer since it treats the main question, althougb the last part of the question is still pending, but its ok – Nikos M. Apr 1 '14 at 19:50
regarding the coin tossing landing same side, the question has the intention of interplay between probability / entropy and a physical system, as far as the mechanics are concerned the relative probabilities are not of the essence here (fair coin was used as simple example, if probabilities are 1/3, 2/3 does not matter), on the other hand if ther is a certain deterministic process that generates sequences (maybe with some error margin), then it can be studied by other means – Nikos M. Apr 1 '14 at 20:31
"they dont point to the main question" Which is the "main question"? – leonbloy Apr 1 '14 at 20:48
@leonbloy, i posted comments for various ways to view the question regarding entropy/information/probability in your answer and tried to highlight the interplay between these concepts – Nikos M. Apr 1 '14 at 21:09

## 3 Answers

(1) is correct. A coin has no memory and does not know that it has already come up heads (or whatever) a lot.

Your final question is a case of statistical hypothesis testing. If you search Google for "hypothesis testing fair coin" you will get lots of hits.

-
thanks for the answer, but it does not explain why the second option/reasoning is not correct or not applicable, since entropy holds for sequences of memory-less systems, the question already notes this point, regarding hypothesis testing, yes it is related i opted to use simpler probabiklity terms – Nikos M. Apr 1 '14 at 19:52
a related question if that helps to enhance understanding is trying to figure out how random a (black box) random genarator is, this is very similar although it is phrased as a gambler's fallacy (or maybe not :)) – Nikos M. Apr 1 '14 at 20:03
see updated answer which addresses all the points in the question and the answers given – Nikos M. Sep 27 '15 at 13:21

David answer is right. Regarding your second (wrong) alternative, related to "typical" sequences: the point is not that $n$ is finite. Imagine you have observed 7 coins, with 4 heads and 3 tails. If the next is tail, you would have the same number of tails and heads (the sample average would coincide with the true mean); true, we know (law of large numbers) that the sample average should tend to the mean; true, we know (binomial distribution) that it's more probable to have 4 tails in 8 coins than to have 3 tails. At the same time it's (exactly) equally probable that the next coin is tail than head. And a particular sequence with 7 heads and 3 tails followed by a tail is equally probable as the same sequence followed by a head. Say: HTHTHTHT has the same probability as HTHTHTHH.

-
yes you see this is the crux of the question, all sequences are equally probable so for a fair coin an infinite sequence of just HEADS is equallu probable in this sense (of pointwuise events) but as a sequence its probabbility tends to zero (regardless of memory as stated in previous answer) – Nikos M. Apr 1 '14 at 19:55
so the last part of the question is related to an estimation of when (at which n, depending on chosen threshold) should we assume that a coin is NOT fair, the reasoning for this is given in the question, it is true that it can be seen as hypothesis testing, but i would like to view it through the probability/entropy concept and their inter-play on systems – Nikos M. Apr 1 '14 at 20:01
"an infinite sequence of just HEADS ... its probabbility tends to zero" But the same is true for an infinite sequence of alternating HTHTHT.... – leonbloy Apr 1 '14 at 20:15
"you are wrong" Care to say why? Both sequences are equally probable. Entropy is irrelevant here (especially for the case of the fair coin, where ALL sequences are typical). – leonbloy Apr 1 '14 at 20:45
I know what is a typical sequence. I repeat: a HHHHHH sequence is as typical (in the entropy sense) as HTHTHT.... if p=1/2 . Because they have the same probabilities. Enough for me. – leonbloy Apr 1 '14 at 21:32

Ok, i would not like to answer my own question but i will do it.

You see the crux of the question is this:

While each elementary outcome may have the same probability ($\frac{1}{2}$), the probability of a sequence of outcomes, which is the product of the probabilities of each elementary outcome, which, in this case of a fair coin, means that a very long "a-typical" sequence of just HEADS has the same probability of a "typical" sequence where the relative frequencies reflect the elementary probabilities.

Some may think that this is correct, and entropy does not play a part here.

i will try to show that this is not so.

First an elementary remark, The theory of probabilities has a meaning for events and systems that exhibit "statistical stability" for the duration of certain study or experiment.

The fact that the probabilities of boys/girls tend to be $\frac{1}{2}$, is NOT an inference of a theory, but the result of observation and extensive research through statistical archives related to rate of births. It is not that since $2$ elementary events are present we assign $\frac{1}{2}$ to each. A simple counter (albeit exaggerated) example is when someone falls from a tall building. Two elementary outcomes are present, either the person lives or dies. However the symmetries of the problem and the obesrvation data do not lead to $\frac{1}{2}$ probability for each event.

1. Consider the case where the HEADS / TAILS events have probabilities $1$ and $0$. In this case one would expect any sequence of events to have a certain form and not others (meaning the relative frequences should reflect those elementary probabilities). Similarly if the H / T probs are $\frac{1}{3}$ and $\frac{2}{3}$, one would still expect the sequences to have certain forms and not others. The confusion starts to occur when the H / T probs are equal and a confusion around the maximum entropy distribution as the most un-informative. Well this is NOT correct. A completely un-informative distribution is just this: $P_H + P_T = 1$ (no other information, all forms go). But $P_H=P_T=\frac{1}{2}$ is NOT un-informative in this sense, it states that BOTH outcomes ARE present and by the (more-or-less) SAME amount.

2. Conversely, this is just how one could infer/estimate a distribution from a sequence. Observing a very long sequence of just HEADS, one would not infer that $P_T=P_H=\frac{1}{2}$ (why not? since with these probs this "a-typical" sequence has the same probability)

3. Changing the domain a bit from probabilities to formal languages, and considering the previous points, we can see that each assignment of probabiltiies to elementary events HEADS / TAILS (eg $P_H=P_T=\frac{1}{2}$) implies a language $L_P \subseteq {\{H,T\}}^*$ where $\left|P_T-\frac{1}{2}\right| < e_1$ and $\left|P_H-\frac{1}{2}\right| < e_2$. Where $e_1$, $e_2$ are positive thresholds/margins

4. Which brings us to the next topic, that of ergodicity and "typical" sequences. Some might object that the "law of large numbers" has meaning only towards infinity and will not make any observational difference. This is NOT correct. Let me explain. There is actually a law of "small numbers" (which can be seen as a variation of Tchebychef's theorem and Shannon's theorem). To understand this better, ask at what point does the law of large numbers start to take effect? It should be there at any point (within thresholds). The law of small numbers is this: For every sequence of size $N$ with elementary probabilities $P_H=p_1$, $P_T=p_2$, $p_1+p_2=1$, the relative frequencies, as derived from the sequence, should reflect the elementary probabilities within a given threshold (which might depend on $N$)

In other words for each $N$, $\left|F_H - P_H\right| < f_1(N)$, $\left|F_T - P_T\right| < f_2(N)$, where $F_H$, $F_T$ are the relative frequencies of the elementary events as reflected in the sequence and $f_1(\cdot)$, $f_2(\cdot)$ are functions of $N$ which give the margins/thresholds of accuracy.

In this light the "law of large numbers" states that in the limit towards infinity this relation is EXACT (margins are ZERO). Because at each step an approximation of the law of large numbers holds (a related result is the series of fluctation theorems).

Consider this counter-example, since infinity + infinity = infinity. Why should we not observe a very long sequence of just HEADS (considering $P_H=P_T=\frac{1}{2}$) and all the "missing" TAILS come after that?

A rough answer to this counter-example is that it is equivalent to the throw of $2$ different coins with different probability distributions (ie one coin with $P_H=1$ and one coin with $P_T=1$). However the entropy of the "fair" coin is $1$ while the entropy of each of the other coinds is $0$. And since the $Ent_{fair} = Ent_H + Ent_T$ (since for this counter-example they are equivalent) we have $1 = 0$. (This is a variation argument on the entropy as an invariant of isomorphic shifts, for example Ornstein's theory)

Finally this means that $P(X_n | X_{n-1}X_{n-2} \dots) \ne P(X_n)$ in general, (the $X_n$ outcome of the sequence "depends" on other outcomes, this is how the law of "large numbers" and "small numbers" can hold). NOTE that statistical independence is not a requirement for the (strong) law of large numbers to hold, see for example:

[update]

To further investigate the issue (and also address already given answers on this post), there are two (or rather three) DIFFERENT experiments which are NOT isomorphic, but conflated, nevertheless, in statistical/probability/mathematical literature etc..

a. Throw, in sequence, $n$ times SAME SINGLE COIN

b. Throw, either simultaneously or in sequence, $n$ IDENTICALY DISTRIBUTED DIFFERENT COINS ONCE EACH

c. Perform, in sequence, $n$ times the EXPERIMENT b.

a and b are NOT isomorphic. Actually for b a "sequence" of $n$ tails (or heads) is not problematic at all, BUT for a it is (see for example points 1,2 above).

a and b are HOMOMORPHIC, and entropy makes this explicit and precise, in the sense that the space of outcomes of a is identified ONLY WITH A SUBSPACE of outcomes of b, the "typical" sequences described by entropy (which is an invariant, and LLNs, which describe TIME-averages)

In the same sense, c is only HOMOMORPHIC to a subspace of b exactly because by a, EACH SEPARATE COIN SATISFIES ITS ENTROPY (invariant)

These subtleties (of the 3 experiments) have not been addressed/described/explained before (search in literature/texts/etc..)

[/update]

Is there a way to formalize this "weird" relation while at the same time simplify computations?

And this is the most interesting part.

What quantum mechanics does, is just this, it treats Probability Amplitudes as independent (they can be added as needed) BUT the Probability is the SQUARE of a probability amplitude (so the dependencies, eg "phase information" in quantum mechanics, is taken into consideration)

[update]

The above analysis of the 3 experiments, highlights the analogy (rotation) between Quantum Mechanics and Statistical Mechanics; in this sense QM corresponds to an analogy between a-b, and SM to an analogy between c-b, experiments respectively (see also http://physics.stackexchange.com/a/107436/44176)

[/update]

UPDATE: A related post on exotic probabilities which can be relevant in this discussion

-