# Countable vs. Uncountable Probability

In Apostol's Calculus II, he splits the calculus of probability into two chapters:

1. Finite and Countable sets
2. Uncountable sets

He only remarks that explaining why different tools are needed for uncountable sets "would take us too far afield."

I have an intuition for why a probability measure defined over a finite set would have different requirements than one defined over an infinite set, but I'm not really sure why uncountable sets are so different.

I suspect it may have something to do with the first theorem of the chapter (if $S$ is an uncountable set, there are an most a countable number of $s\in S$ where $P(s)\not=0$), but I'm not sure what exactly.

A search online doesn't seem to turn up much, so maybe this is just a decision the author made to separate them that others don't agree with? More likely I just don't know the terms to search for.

-

Let us suppose first that the sample space $\Omega$ is finite or countably infinite. Then $\Omega=\{\omega_1, \omega_2, \cdots, \omega_n\}$ for some positive integer $n$, or $\Omega=\{\omega_1,\omega_2,\omega_3,\dots\}$. In either case, if we know the $p_i=P(\omega_i)$, then for any subset $S$ of $\Omega$, we can find $P(S)$ by summation (finite summation, or summation of an infinite series).

Suppose now that $\Omega$ is uncountable. Then, as you mentioned, there are at most countably many $\omega$ such that $P(\omega)\ne 0$. We can separate out these, but will be typically left with an uncountable subset $\Omega' \subseteq \Omega$ such that $P(\Omega')\ne 0$. Indeed in the majority of standard situations, $P(\omega)=0$ for all $\omega$. For uncountable subsets $S\subset \Omega$, we will therefore not be computing $P(S)$ by a summation. In the usual situations (normal distribution, exponential distribution, many others both univariate and multivariate), computations of probabilities and related quantities such as expectations are done by integration.

So there is a large difference between the finite or countably infinite case on the one hand, and the uncountable case on the other, in the tools that we use. Typically it is summation versus integration. One could say that summation is a special form of integration. But that does not make the distinctions collapse entirely. Typically, for technical reasons, not all subsets of $\Omega$ can be assigned probabilities in the uncountable case.

-
Is there a simple example of an $f$ where $\forall x\in [a,b]:f(x)=0$ yet $\int_a^b f(x)dx\not=0$? – Xodarap May 6 '12 at 18:33
There is no simple example, indeed no example at all. Can't happen! But imagine a piece of wire of length $1$, where the density of the wire is say $2x$ at $x$. The "weight" of a point is $0$, but the total weight of the wire is $\int_0^1 2x\,dx=1$. Continuous probability distributions work like that. Discrete distributions (finite or countably infinite) can be thought of as point masses at specific places. Continuous ones are smooth distributions of mass. (There can be hybrids, but they don't come up often in applications.) – André Nicolas May 6 '12 at 19:03
Or else use area under a curve as an analogy, for example the area under $y=e^{-x}$, $0$ to infinity. The region is "made up" of uncountably many vertical line segments. Each has $0$ area, but together they have area $1$. – André Nicolas May 6 '12 at 19:12
I see. It's misleading to talk about $P(x)$, it's kind of better to consider it as an area $P(x)*dx$ – Xodarap May 6 '12 at 22:36
Not misleading, just not useful since $P(x)=0$. We have similar situation with rate of change. As $x$ changes from $3$ to $3$, or $7$ to $7$, $x^2$ changes by the same amount, namely $0$. But the instantaneous rate of change (derivative) is different at $3$ than it is at $7$. – André Nicolas May 6 '12 at 22:42

One essential difference is that it is possible in some cases to sum a finite or countably infinite number of positive real numbers and have a result of $1$. So these numbers can be probabilities of disjoint events.

It is never possible to do the same with the sum of an uncountable number of positive real numbers.

-