Chance versus Skill 
Question. How does one mathematically analyze situations that involve chance and skill?

Let's take the coin flip as a simple example. Assume that it possible to skillfully flip a coin to get the landing you want. Also assume zero cheating.
FIRST SCENARIO
The world's 5 most talented coin flippers gather to compete. The results:

Person 1

*

*Coin Flips: 100


*Success Rate: 100%
Person 2

*

*Coin Flips: 10,000


*Success Rate: 90%
Person 3

*

*Coin Flips: 1,000,000


*Success Rate: 80%
Person 4

*

*Coin Flips: 100,000,000


*Success Rate: 70%
Person 5

*

*Coin Flips: 10,000,000,000


*Success Rate: 60%

Each person claims he is the best coin flipper. How would you analyze the results?
SECOND SCENARIO
A man claims he is so skilled at coin flipping, he can always land heads.

He flips one coin. Sure enough, heads.
He flips again. Heads again.
He flips 100 times. All heads.
1000 times. Still all heads.
After 100,000,000,000,000 flips, every single one heads, he stops and
says "I told you so."

When do we go from thinking "He's lucky!" to "He's good!"?
 A: One other reasonable approach is to accept the "statistics dogma", which is, roughly speaking, that "the predefined events of probability more than $\alpha$ may be due to chance and the rest is due to cause". It is traditional to set $\alpha=0.05$, but this value is in no way sacred. So, if the coin flipping skill is measured by the probability $p$ of landing heads, to convince you that his head landing skill is at least $q$, the person needs to announce and then achieve the result that would have probability less than $\alpha$ for any flipper with skill $q$ and below. From this point of view, the perfect result in $100$ tosses (if announced beforehand) convinces you only of the skill $\alpha^{1/100}$, which for $\alpha=0.05$ is about $0.97$. The pre-announced perfect result in $10$ tosses convinces you only of skill $0.74$, etc.  
A: The second question should come first :

SECOND SCENARIO : 
  When do we go from thinking "He's lucky!" to "He's good!"?

Since you deal with probabilities, any number of results could be explained by luck. The most general way to sort this out is to compute what would be the odds of such a result happening by pure luck. The statistical approach is to choose a metric (here the success rate is pretty easy to choose), and then to compute the probability of the hypothesis "the result is due to luck". This probability is the p-value. Below a certain treshold, you will state the result is due to skills. 
In the case of a coinflip, you would look at the probability that the guy lands all these heads with a 1/2 probability of landing heads. 
The p-value is widely used, with varying values that depend on how sure you want to be not to select false positives. 
Because that's the trick : Whatever your method of selecting "truly good" coinflippers is, you will always be wrong some times. What's even worse is that if you want to know your actual probability to be wrong, you need to make extra asumptions, because to do this (by using bayesian inference), you need to the actual probability of your hypothesis to be true. 
You can increase as much as you want your ability to find "truly good" coinflippers from "lucky frauds", but you cannot know how likely your guy is to be in the first group (which is the question you want to answer), unless you assume the actual repartition of people between the two groups. p-value only gives you the probability for you to be wrong in the case the contestant is a fraud. If you already knew how many skilled coinflippers there were and how their skill is distributed, you wouldn't be holding this giant coinflipping contest.
This sheds a light on the difficulty to answer your second question in the general case.

FIRST SCENARIO : The contest

It all boils down to the definition of what is "being a good coinflipper" ?


*

*Is it showing that you can actually throw a fair coin with a probability to land heads different of 1/2 ?  In that case you would use the p-value of the hypothesis "p(heads)=1/2". The last contestant would win in this framework, but all of them have incredibly low p-values.

*You can estimate the actual skill, but you need to make asumptions on the actual distribution of skills. For instance, if you assume it is possible to achieve a 60% success rate by knowing how to throw the coin, you would set different expectations. But for this, you need additional hypothesis, I won't go into the details of all the possibilities
I think a good way to organize such a contest would be to request people to show the highest Lower bound of confidence interval for a Bernoulli parameter, as suggested here.
In nearly plain english,  is to define some (very low) confidence level alpha, and to set the score of each person that participated as "The lowest possible actual probability of success of the coinflipper that would have given such a good record with a probability higher than alpha"
I like this solution because any contestant will eventually converge towards it's actual success rate, but will very rarely (with a small probability of your choosing) be above during its convergence process. Thus any contestant patient enough can hope to overcome any opponent in the end, knowing that this opponent is very unlikely to have been ranked to high out of luck.
Let's choose alpha as 0.01%
This method enables us to select a winner for your contest : Person 2 wins !! with a score of 88.8%
Note that with a p-value of 1%, Person 1 wins, and that my poor Excel overflows before I reach a probability that will make person 3 wins, probably around the event of a 27 sigma variation in a normal distribution.

My solution is not perfect, because as small as I can chose my alpha, there is always the possibility that some contestant gets lucky enough to get a lower-bound estimate too generous, which would allow him an advantage that could not be overcome by other contestants that can only hope to converge towards their true success rate. Depending on the hypothesis you are ready to take, and what you want to achieve, you will choose one or the other
A: Starting from the second person, you are in range of the normal approximation to the Bernoulli distribution, so the standard deviation on the number of heads is $\sqrt {10000\cdot 0.1 \cdot 0.9}=30$ so we can say his probability of heads is $0.90$ with a standard deviation of $0.003$  For the others, the probability of heads is measured even better, so we would say the order of skill is from the top to the bottom.  For your second, there is no sharp line where we say the transition is from "might be luck" to "must be skill".  At each step you can compute the chance that the heads are all luck.  It is judgement where to say it is unreasonable to consider it luck.
A: With no "skill" a person would flip heads half the time and tails half the time.  If such a person flips a coin 100 times, the probability of getting "all heads" or "all tails" or any specific pre-designated result for all 100 flips, whatever you mean by "100% success, is $\left(\frac{1}{2}\right)^{100}= \frac{1}{1267650600228229401496703205376}$, about 0.000...0007888 where there are 31 "0"s after the decimal point.  That's awfully small!  How small you consider evidence of "skill" is up to you.
A: The simple, high-level answer to your question, is that we never flip a switch in our beliefs from thinking "he's lucky" to "he's skilled". Instead, we assign both of these alternatives some measure of belief, some guess at how likely they are to be true, and then we adjust our beliefs as we observe his performance. Depending on what beliefs we started with, certain observations will make one or the other hypothesis seem the most probable at any given moment – but still further observations could tip the balance back the other way.
In these examples, we have a good model of how likely each outcome is to occur by chance. But you can't draw any conclusions just from the probability or improbability of the situation you find yourself in without some idea of how likely each outcome is to occur by skill.
Roughly speaking, if you believe it's easy to be skilled in the art of coinflipping, it seems more likely that Person 1 achieved their perfect ratio by skill than luck, so Person 1 is probably the best coinflipper, and Person 5 is the worst.
If, on the other hand, you believe it's hard but possible to flip a coin in a way that turns up heads 80% of the time, but basically impossible to do better than that, then Person 1 and Person 2 are clearly just lucky, and Person 3 is probably your best bet, since they hit the maximum attainable accuracy over so many trials that it's extremely unlikely that they're actually just a lucky 75%er, whereas Persons 1 and 2 haven't proven that so thoroughly.
Now, if you find it extremely difficult to believe that anyone can do better than chance on a coinflip, and you believe that getting 52% heads is astronomically harder than 51% heads and so on, then even though the results of People 1–4 are fantastically unlikely to arise by chance, they're even more unlikely to reflect the skill of the cointosser, and so you look to Person 5 as the person whose success is hardest to explain by sheer luck.
Finally, if you're perfectly 100% confident that the players had no influence on the outcomes of the coinflips whatsoever, it simply must be the case that they're all absurdly fortunate, and all charlatans.
