Which biased coin is better? My friend has 2 biased coins. He flips the first one a number of times
and tells my how many heads and tails he saw, then repeats this for
the second one. I have no influence on how many trials he does, though
I am guranteed at least one per coin. My job now is to tell him which
coin is better; i.e. which one has a higher probability of showing
heads.
The obvious way to do this would be to estimate the probabilities from
the samples, however this seems wrong to me because I am not taking
into account how accurate my estimates are. E.g., consider
the case where:


*

*Coin 1 has shown 501 heads and 499 tails.

*Coin 2 has shown 2 heads and 1 tails.


I tried approaching this in a few ways, and all of those quickly
showed me how little I know about probability theory and statistics.
My question has four main points:


*

*Does this problem depend on how the coin probabilities are distributed? (If so, assume a uniform distribution.)

*How do I calculate how likely either answer is correct or wrong, given the two samples?

*Is there a strategy that is more often correct (or less often wrong) than the obvious one?

*How does this generalize to any number of coins?

 A: Let me try to give you an intuitive answers for the first two of your questions. 
Q1: Does this problem depend on how the coin probabilities are distributed? (If so, assume a uniform distribution.)?
A1: Yes, the problem depends completely on the distribution of the probabilities becuase once you know this information you can completely determine the probability of any specific scenario. The assumption about the distribution to be uniform is very strong. Say your coin is either such that it lands on "head" or "tail", or, it is such that it can land on "head", "tail" or on the "edge". In both of these cases, the uniform probability means that the probility will be distributed "equally" into all possible events, i.e. in the first case we would have probability $1/2$ for both "heads" and "tails" whereas in the second case the probability would be $1/3$ for any of "head", "tail" or "edge". So we see that the assumption determines the probability distribution if we know what kind of coin we are dealing with. To sum up, we shall not assume the probability distribution to be uniform because it would be contradictory with the initial assumption that the coins are biased (meaning precisely that the distribution shall not be uniform). 
Q2: How do I calculate how likely either answer is correct or wrong, given the two samples?
A2: This is a good question since trying to determine which coin is better amounts to approximate the "real" probability distribution of the process. This is interesting because in this point we should ask ourselves  what actually the statement something has probability equal to p means. Say we have a coin such that it definitely either lands on one or the other side and we know that the coin is biased. Moreover, assume that we know the probability distribution of the coin flipping: the chance that the coin lands on "head" is $1/3$ and that it lands on "tails" is $2/3$. Then what this actually says is that if we produced a large number $n$ of flips, then the number of "tails" we have registered is approximately two times larger then the nuber of "heads" and if we do further flipping we would observe that this approximation is better and better. This means that in the limit $n \rightarrow \infty$, the ratio "tails"/"heads" would approach number $2$. From this consideration, the sample that is larger contains a better information from which you can predict the probability distribution of the coin flipping. This is in correspondence with your intuition that you would not take into account the accuray of your estimate if you would see the two samples, one consisting of 501 + 499 data samples and the other one only from 2+1, as equally "valuable". Naivly, the first sample would say that that the probability of "head" or "tail" seem to be close to $1/2$ whereas the second sample would say the probability could be $2/3$ and $1/3$, respectively.
Hopefully this helped a bit :)
A: Well, let me address your points in order:


*

*Yes, it crucially depends on how the coin probabilities are distributed. Indeed, what you need is the joint distribution of both coins. For example, you might assume that while the single-coin biases are uniformly distributed, your friend has explicitly chosen two coins whose bias goes into opposite directions. That would be a completely different situation than when the biases of the two coins are independent. For example, in the first case, if you had one coin with 1 head and 2 tails, and one with 120 heads and 200 tails, then you'd clearly select the first coin, as the second one has shown without doubt to be the one biased to tails, so the other one has to be the one biased to heads, even though its few results hint at the opposite. While in the second case, the clear bias of the first coin would, of course, tell you nothing about the bias of the second coin (that's exactly what independence means).
In the following, I'm going to assume independence in addition to uniform distribution for each coin; for example, your friend might just have taken two random coins from his big collection of biased coins.

*If we label the two coins with A and B, the relevant question here is: What is the probability that coin A is the one more likely to show heads than coin B? If that probability is higher than 50%, you should choose coin A, otherwise you should choose coin B.
To get the probability, we need to use Bayes' rule. Bayes rule can be states as: The probability of A given B is proportional to the probability of B given A times the prior probability of A. The proportionality factor then can be derived directly from the requirement that the probabilities must add to 1.
Now using the binomial distribution, the probability of a coin with heads probability $p$ to land $h$ heads and $t$ tails is $\propto p^h(1-p)^t$. Following Bayes, so is the probability of a coin having a heads probability $p$. Since we assume an uniform prior, this amounts just to a proportionality constant that will cancel out in the end.
Since we assumes the two coins to be independent of each other, the joint probability distribution of both coins' probabilities is just the product of the individual probability distributions. Now to get the probability that coin A is the better one, we just need to integrate over the subspace where $p_A>p_B$, and divide that by the integral of the total distribution for normalization:
$$P(A\text{ is better}|h_A,t_A,h_B,t_B) =
\frac{\int_0^1 \mathrm dp_A \int_0^{p_A}\mathrm dp_B\,
p_A^{h_A}(1-p_A)^{t_A} p_B^{h_B}(1-p_B)^{t_B}}
{\int_0^1 \mathrm dp_A \int_0^{1}\mathrm dp_B\,
p_A^{h_A}(1-p_A)^{t_A} p_B^{h_B}(1-p_B)^{t_B}}$$
For some reason Mathematica 8 doesn't want to symbolically evaluate this expression (it claims for integer $t_B>0$ it were undefined), but if I insert the numbers of your example, it gives me
$$P(A\text{ is better}|h_A=501,t_A=499,h_B=2,t_B=1) = \frac{21126}{67201} \approx 31\,\%$$
So in that case, you're about twice as likely to choose correctly if you choose coin B over coin A (assuming the assumptions on the probability distributions are justified).

*I doubt that there is a better strategy than simply choosing the coin with the higher frequency of heads (I could say for sure if I had an analytic expression for the fraction of integrals above).

*Generalization of the above calculation to an arbitrary number of coins is straightforward, except that now one number is not enough to decide whcih coin is best; you'll have to calculate the probability of the coin being the best one for all the coins separately (well, you can omit one, as you know the sum of all probabilities must be $1$).
