What does probability really mean?

Question

I apologize in advance if the question is a bit off-topic and not strictly mathematical.

To be clear, I'm talking about classical probability which is defined like this: Given a finite sample space $S$, and a subset of $S$ which we call event $E$, the probability $P(E)$ of event $E$ is $$P(E)=\dfrac{N(E)}{N(S)}$$

Where $N$ denotes the number of the elements of a set.

I wanna understand what notion we try to capture when we define probability. Bernoulli, Laplace and others certainly had a certain concept in mind that they wanted to describe and hence they formalized it into this definition.

To explain my point, consider the following example: We have a set of data, say, a bunch of real numbers and we want to represent them using a single real number. So the notion we wanna capture here is finding a real number that is the best representative to a group of real numbers. It turns out the mean of those numbers is the best representative to them in the sense that it minimizes the sum of the square of differences(distances) between the mean and every real number in that set of data.

So what notion we wanna capture by defining probability? I was told that if the probability of an event $E$ in an experiment is say, $0.3$ then if the experiment is performed $n$ number of times, $0.3n$ of times event $E$ will happen. So this is the notion $P(E)$ supposed to capture.

This definition, as far as capturing this concept makes perfect sense to me in the following:

-If an event has a $P(E)=0$ then it will never happen.

-If an event has a $P(E)=1$ then it will always happen, or if we perform the experiment $n$ times, $E$ will occur $n$ times.

However for $P(E)$ that has a value between 0 and 1, I don't know how it works. For example if we tossed a coin 10 times, it's not guaranteed at all that half the tosses will give you head, even worse, all the tosses can sometimes give you tails. So what's going on here?

When I asked for the justification that $P(E)$ really captures this notion, I read this is justified by the law of large numbers: You perform an experiment certain number of times, say for example tossing a fair coin. When we calculate the probability of a toss being head, we assign the number $0.5$ to that event. This means(According to my understanding of the law of large numbers) that as the number of tossing the coin gets arbitrarily large(number of trials approaching infinity), Heads will make up $0.5$ of the total number of coins tossed, or a number that's very close to $0.5$ and as the number of trials increase, it will approach $0.5$.

However there's something circular about this: When I ask what does "fair" coin mean? It's a coin that, as we toss it arbitrarily large number of times, the number of its heads will approach $0.5$.

So can someone clear it up for me, and justify why $P(E)$ captures what it captures?

Answer 1

When we talk about probability, we usually want to express the expected outcome. So since on a large scale, a coin will probably land heads 50% of the time, and tails 50%. The probability of a coin being radically off this point grows closer to 0, until we toss a coin infinity times, where the chance tails will be extremely different from heads is 0. Probability is the liklihood that something will happen. It is not intended to predict outcomes. It is not supposed to tell you what will happen, however it usually is quite accurate on a large scale (in other words, on a large scale, there is a higher probability of probability being the outcome.) In real life, if you are one of those people who believe that the world is set, then in real life, everything has probability either $1$ or $0$. God either destined it to happen, or God decided it won't happen. However, in a purely mathimatical world, we can use probability as the outcome, which is useful in expected outcome (where the outcome is the probability, for example, every time you roll a die, you get 3.5), and gambling (you assume that the probability of your winnning * the reward is what the outcome will be), because given a large enough scale, probability will probably reflect the outcome. Although it doesn't have to.

Hopefully I answered your question, if I didn't, please comment and I will try to fix my answer :)

Answer 2

I can try to put it simply in non-matematician language. If I don't get namely what you wanna ask, write in comments, we'll try to clear it up together :)

About the law of large numbers here: as far as I get, the more events happen - the less randomness occur. Let the "fair" coin (i.e., without any deformation) has the first side (head) and the second (tail). We infinitely toss it and finally have some percentage of "heads". Now let's pretend that the first side was not "head", but "tail". The result should be the same, right? Because it's the identical thing, just with another name. So the only percentage that satisfies here is 50% or 0.5. But if the tosses are finite, of course, some randomness occure and it's not always 0.5, but still the more you toss - the closer probability gets the value for infinite tosses. And if to say about a single toss, there are only two ways ("head" and "tail") and both of them are identical (changing the names shows it) so we divide 1 (outcomes of a single event) by 2 (possible outcomes) and get 0.5.

Another example. There is a cage where there can be either a lion, or a wolf. Or none. Maximum possible events=3. So the probability of seeing the wolf today is 1/3. And of seeing the wolf or a lion=2/3 because there are 2 events that satisfies us (seeing a wolf or a lion). But the probability of seeing the wolf AND the lion is 0/3=0 because there is no possible event when they are together in the cage. Let's make it more complicated. There is a statistics that during the last year every calendar week wolf was in the cage 4 days a week and lion respectively 2. And one day a week the cage was empty. It means that for the whole time the wolf was there 2 times more often than the lion and the lion was 2 times more often than the cage was empty. So if the probability that the cage is empty today equals to x, the probability of seeing the lion=2x and of seeing the wolf is 2*2x=4x. 4x+2x+x=1 so x=1/7, "wolf's" probability is 2/7 and "lion's" probability is 4/7. If calculating it in terms of a single week (not the whole period) the result is the same: 7 days total, 4 days of 7 total are for lion, etc. If to modify this example again and state that the cage is empty only on Sunday (not the random day a week, as previously stated) so if I go there on Saturday the probability of seeing the lion is 4/6=2/3. Because Sunday doesn't count in this case. And if I come there on Sunday the probability of seeing empty cage is 1 (and respectively 0 for lion or wolf). That's what probability means.

To be honest, probability is not a precise thing, the above examples are very artificial. The zoo watcher could get ill and for a whole week the cage could remain empty. Or there could be wolf there all days because the watcher fell ill right after the day when the wolf was there. This is the random factors. And P(E) means the likeness (but not states that it will definitely be this way).

Answer 3

The definition you gave is a special case of the standard measure theoretic one when restricted to finite sample spaces. In the general but still finite case, you would have a weight for each element of $S$ that you'd sum. As far as mathematics is concerned, that's all there is to it. This is why people are talking about "philosophy" and even you seem to acknowledge that this is "not strictly mathematical".

Since you reference Laplace, you can read his views in A Philosophical Essay on Probabilities. In particular, the second section, "Concerning Probability", introduces his views. It's quite clear from that that he is a determinist (i.e. he believes there is only one actual outcome to any scenario) and that probability only arises due to our ignorance. In modern terms, he's undoubtedly a Bayesian. I'd have to look it up, but I'm pretty confident Bernoulli and many of the early founders of probability theory were Bayesians.

Looking at why we say a coin toss has probability 1/2 from this perspective, I think is enlightening, especially in the following thought experiment. Let's say you trust me completely, and I tell you a coin is "biased" so that a coin toss returns either heads or tails 90% of the time, but I don't tell you which of heads or tails it is. I'm about to flip the coin, what is your probability that it will come up heads? It's 1/2, at least that's what Laplace or a Bayesian in general would say, even though you know that if flipped "many times" it will not come up heads or tails in about even proportion. You can even take it to the extreme if "biased" unsettles you (which it should a little): instead of 90%, I say 100%, e.g. it is a coin with two heads or two tails, say.

How do you or Laplace get 1/2? It's from symmetry. If my beliefs about the result of the coin toss don't change when I swap "heads" and "tails" in my prior beliefs, then I should assign equal probability to statements which differ only by swapping "heads" and "tails". This is Laplace's "Principle of Indifference" or "Principle of Insufficient Reason". In the scenario above, the main piece prior information was "I tell you a coin is 'biased' so that a coin toss returns either heads or tails 90% of the time" which is clearly equivalent to the statement with "heads" and "tails" swapped. That said, if you had additional prior information that would break symmetry - maybe you know I have a freakish aversion to faces, you would not be able able to use the principle of indifference.

To a Bayesian, the intuitive notion to which the mathematical notion of probability corresponds is "degree of belief". Of course, it doesn't make sense to talk about my "degree of belief" in a set. Instead $P(A|I)$, the probability of $A$ given $I$, is defined for propositions $A$ and $I$. We can arrive at the rules of probability by formulating a set of functional equations driven by "rules of rational reasoning". These functional equations have a unique solution (up to monotonic rescaling) which is $P$. This result is Cox's theorem. Then, to get the version you have, we let $S$ be a set of independent propositions and $P(E) \equiv P(\bigvee E | I)$ where $I$ is the fact that the propositions in $S$ are independent and at least one holds, and $\bigvee E$ is the disjunction of the propositions in $E$. If the background information says nothing further about the propositions in $S$, then, by the principle of indifference, they have the same probability and we recover the counting formula you had. The Bayesian perspective makes introducing a "sample space" of equally probable propositions from which all other (relevant) propositions can be derived unnecessary and unnatural.