# Guessing the probability by results of just 1 experiment

I have a probability question that seems easy, but I somehow can't wrap my head around it.

Suppose we have a coin. Probability that coin toss will come out heads is some unknown value X. First toss came out as heads. What would you be your best guess about the value X (so, if you guess is y, your task is to minimize $|X - y|$)?

For me it seems like given the result of the first experiment, coin is just a little bit more likely to be loaded in a way that heads come out more often, so optimal guess about the likelihood of heads is 1. But I can't formulate it in a proper way or prove it mathematically. Besides, there is an opinion in other (non-math) online community that probability 0.5 would be more likely. I think there is a flow somewhere in my logic.

Can you help me to understand this concept? Thanks.

Update: for anyone interested, the question originally emerged during the discussion of Hindsight bias phenomenon. More precisely, the result of Fischhoff and Beyth experiment seems to be logically correct since differences in the results of predictions were caused by the differences in the information given to the groups. Even if the students were explicitly asked not to consider the result of conflicts as the probability factor, the only thing that experiment states is that we can't throw things out from our subconscious perception of the world at will (and that is obvious from the definition of the subconsciousness itself). So the phenomenon of hindsight bias can not be tested through such experiment or any one alike. The experiment should show difference between mathematical probability and empirical probability given the same initial data.

-
If you can carry out just one toss, the only definitive conclusion you can make is $X\ne 0$. –  Bravo Mar 6 '13 at 11:46
Yes, but it also obvious that guess 0.0001 is less likely to be right than guess 0.9999. So there must be some way to calculate optimal guess. –  Scorpil Mar 6 '13 at 11:51
Are you planning to use this result in any practical manner? Theoretically speaking, your best guess is 1, however that would be unlikely in any practical situation, as realistically one would at least attempt to determine a mean based on a reasonable sample size, assuming they didn't already know the underlying distribution –  mardat Mar 6 '13 at 11:58
I have no intention to put it into practical use. It's just a thought experiment. Actually, it came out in the discussion about how human logic works and if it is consistent with mathematical principles in one particular experiment. I'll update the post now to give more intuition about where this question came out in the first place. –  Scorpil Mar 6 '13 at 12:09
You cannot deduce a trend, nor a probability, from a single event. –  Carl Smith Mar 6 '13 at 15:04
add comment

## 5 Answers

Let's do this using Bayesian statistics. Let $p_0$ be the probability distribution over the interval $[0,1]$ describing our initial belief in the likelihood of various values of the unknown parameter $X$. We wish to update this distribution based on the outcome of an experiment in which the coin is tossed and comes up heads with probability $X$.

The conditional probability $P(\mathrm{heads}|X=x)$ of the coin coming up heads, given a certain value $x$ of $X$, is simply equal to $x$. Thus, by Bayes' rule, the posterior probability distribution for $X$, given that we do observe the coin coming up heads, is given by $$p(x) = P(X=x|\mathrm{heads}) = P(\mathrm{heads}|X=x) \frac{P(X=x)}{P(\mathrm{heads})} = x \frac{p_0(x)}{C} = x p_0(x) / C,$$

where the normalizing factor $C = P(\mathrm{heads}) = \int_0^1 P(\mathrm{heads}|X=x)P(X=x)\,dx = \int_0^1 x p_0(x) \,dx$ just scales the distribution so that the total probability mass remains one.

(Note that I'm abusing notation a bit here by treating distributions as if they were functions and blithely conditioning on probability-0 events like $X=x$. All this can be made rigorous, at the cost of introducing some extra complexity, but I won't go into all that here.)

Given a particular prior distribution $p_0$, the posterior distribution $p$ will be fully determined, and we can then obtain an expected value for $X$ by integrating over the distribution $p(x)$ weighted by $x$: $$\mathbb E[X|\mathrm{heads}] = \int_0^1 x p(x) \,dx.$$

In particular, if we initially assume every value of $X$ to be equally likely, such that $p_0(x) = 1$, then the a priori probability $C$ of getting heads is simply $\int_0^1 x\,dx = \frac12$, and the posterior distribution is thus $p(x) = x/C = 2x$, giving us $$\mathbb E[X|\mathrm{heads}] = \int_0^1 2x^2 \,dx = \frac23.$$

Indeed, if we start with the flat prior $p_0(x) = 1$ and observe $a$ heads and $b$ tails, the posterior distribution will be the beta distribution $p(x) = x^a(1-x)^b / \int_0^1 x^a(1-x)^b \,dx$, and the expected value of $X$ will be simply $$\mathbb E[X|a\text{ heads, }b\text{ tails}] = \frac{\int_0^1 x^{a+1}(1-x)^b \,dx}{\int_0^1 x^a(1-x)^b \,dx} = \frac{a+1}{a+b+2}.$$

This simple formula is exactly the same as the rule of succession formulated by Laplace in the 18th century to address the "sunrise problem", i.e. the task of estimating the probability that the sun will rise tomorrow, given evidence that it has done so every day for at least the past 5000 years. Your problem is exactly the same as Laplace's except that, instead of 5000 years of daily observations, you only have one. Thus, the expected value of $\mathbb E[X] = \frac23$ you get is also relatively close to the prior estimate $\frac12$.

-
Great answer :) –  Sh3ljohn Mar 6 '13 at 13:14
That's one awesome piece of math right there. Thank you also for the links. I'll need some time to understand this and sort it out in my head, though. Thanks! –  Scorpil Mar 6 '13 at 13:16
I believe you meant to say that the "normalizing factor" $P(heads)$ should be understood as $\int_0^1 P(\mathrm{heads}|X=x) \ p_0(x)\,dx$ (rather than $\int_0^1 P(\mathrm{heads}|X=x)\,dx$), because at that point you haven't yet assumed a uniform prior, i.e. $p_0 = 1_{[0,1]}$. –  r.e.s. Mar 6 '13 at 17:05
@r.e.s.: Indeed I did. Corrected. Thanks for spotting that. –  Ilmari Karonen Mar 6 '13 at 18:22
add comment

Edit : Thanks to Aant, I was able to fix my reasonning. Should I have deleted it instead ?

Let us assume that the probability $X$ of the coin flipping heads is a random variable uniform in [0;1] (pdf f(t)=1). Let $H$ be the event that the first flip is Heads.

$$P(H\cap (X\leq x)) = \int_0^x t\times f(t) dt = \frac{x^2}{2}$$

$$P(H) = \int_0^1 t\times f(t)dt = 0.5$$

Therefore :

$$P(X\leq x |H) = \frac{P(H\cap (X\leq x))}{P(H)}=x^2$$

Thus naming $Y$ the random variable $X|H$, and $g$ its probability distribution function, we have $$\int_0^x g(t)dt = x^2$$

And therefore $g(x)=2x$. It is now a matter of minimising $|Y-y|$ for $y$, and that is achieved by $y=E(Y)=\frac{2}{3}$

This could of course be adapted to any distribution other than uniform at the beginning.

-
I think you should have considered $P(X=x)$ and not $P(X\leq x)$, but that was a better answer than mine and the others already! Look at Ilmari Karonen's post :) –  Sh3ljohn Mar 6 '13 at 13:17
Yeah I saw the post above, and it's obviously correct, but I would very much like to know why mine is false :/ That's pretty much the only reason why i haven't deleted it yet. It didn't think you could use $P(X=x)$ as it's $0$, so i tend to always use $P(X\leq x)$ (again, not an expert:)) –  imj Mar 6 '13 at 13:21
The problem is that your first equation, for $P(H|X\leq x)$, is wrong. What you have written is the value of $P(H\cap X\leq x)$, so you need to divide by a normalising factor of $P(X\leq x) = x$. Then the rest is correct and you end up with the same answer as Ilmari. –  Aant Mar 6 '13 at 14:33
Thank you so much ! Being pretty new here, could anyone tell me what is the etiquette in his situation, delete or edit ? –  imj Mar 6 '13 at 14:51
The minimum of $|Y-y|$ isn't in fact achieved by $E(Y)$. See my answer. –  Oscar Cunningham Sep 4 '13 at 12:32
add comment

Assuming that all you know is that your coin can either come up heads or tails, with an unknown probability of either occurring, and given that you have tossed the coin once and it has come up heads, the most likely probability that the next toss will come up heads is 1. The opinion that 0.5 is the more likely probability is derived from our own prior knowledge of coins - that 0.5 is the most likely chance of a coin coming up heads or tails. However given what you've said, your statement is correct. Note however that although this is the optimal guess given what you know, it is not necessarily the optimal guess in general, and ignores the small sample size

-
add comment

Essentially for a discrete random variable, estimating the probability of a specific event empirically is commonly done using relative frequencies; that is, dividing the number of occurences of a specific event by the total number of experiments. With this technique, you would obtain a guess of 1 for the event 'heads', and that would be your best guess.

Now, you could do better than that, if you knew more. If you assume to know that the coin only has two faces (only two possible events), and that it seems reasonnable to assume an equal prior for both events, you could begin with a probability of 0.5 for heads, and 0.5 for tails. What to do with the first experiment remains ambiguous however, and depends on how you wrote your estimation model. The more you allow one experiment to affect your guesses, and the more unstable your system will be (the guess will not be very reliable no matter how many experiments). On the other hand, the more experiments you require to affect your prior guesses, and the more stable your system will be, but the slower the convergence: with a very large number of experiments though, your guesses should be fairly accurate.

You can see this more intuitively if you consider that guessing a prior of 1/2 for each event is equivalent to considering two piles of size n. The total number of events that you assume for your prior is 2n; that's the 'strengh' of your prior, the 'credit' you give to your guess in a way. Now the larger n, and the more stable your guess will be, but if your prior was wrong, it will require a lot of experiments to reach the correct guess. Is it clear or not really?

-
Pretty clear thanks. It is obvious that there is almost no information to make any reasonably good assumption. But I think some purely theoretical way to reason why 0.001 worse guess than 0.999, and to calculate how much worse it is. –  Scorpil Mar 6 '13 at 12:41
I speak from my own experience, but I might be wrong; working with probabilities, I learned that really, putting any feeling into the calculations only makes it worse. Stick to the facts; in this case, one way to calculate the goodness is to calulate the distance between your guess and the ground truth (0.5) as more experiments come in. Considering this, 0.001 is simply not worse than 0.999, and to characterize the efficency of your estimation model, you should look at the 'assymptotic' expected value and variance. –  Sh3ljohn Mar 6 '13 at 12:54
add comment

I have a small correction to the other answers.

As other answers have pointed out, doing a Bayesian update on a uniform prior gives a posterior distribution of $p(X|H)=2X$. They then go on to calculate the mean of this posterior. But in fact the question asks us for an estimator $y$ minimising $|X-y|$. This is actually achived by the median of the posterior, which is $\sqrt{2}/2$. The mean is what you'd want if you were trying to minimise $|X-y|^2$.

(The mode also has a characterisation like this, it minimises $|X-y|^0$ in the sense that it is given by $\lim_{p\rightarrow 0}\text{argmin}|X-y|^p$.)

-
add comment