9
$\begingroup$

As I understand it, independence of A and B can be informally established by asking whether learning something about one of those events tells you something new about the other. This must be borne out mathematically, however. For example:

If P(A|B) = P(A), then A and B are independent.

And if P(A & B) equals P(A) x P(B), then A and B are independent.

The above imply that P(B|A) = P(B)

It’s this last statement that confuses me, at least in application to certain cases. For example:

You devise a way to randomly choose a number from all real numbers, uniformly distributed. The probability of the chosen number being prime is 0, given that 0% of the reals are prime. Similarly, choosing the number 2 from the set of all real numbers has a probability of 0. Likewise, the probability of choosing a 2 from the set of all prime numbers has a probability of 0. Given that 2 is a prime number, it seems that choosing a 2 and choosing a prime number must be dependent events, at least in one direction (namely, if I know I’ve chosen a 2, then I’m certain I’ve chosen a prime number). Here’s what I mean:

P(2|prime number) = P(2) = 0 (passes for independence)

P(prime number|2) = 1 (i.e., not 0, or P(prime number), and so fails for independence)

But can also test as follows:

P(2 & prime number) = P(2) x P(prime number) = 0 P(2 & prime number) = P(2) x P(prime number|2) = 0 P(2 & prime number) = p(prime number) x P(2|prime number) = 0

Everything here comes out to 0, as I suppose it should. This also aligns with my understanding that anything with probability 0 is independent from any other event. (Right?) And yet, I’m stuck with the intuition that:

If I learn I got a 2, I know I got a prime number, wherein learning I got a prime is insufficient for updating my beliefs about getting a 2 (provided I really do believe that the probability of pulling a 2 from the primes is 0), and the same goes for learning I got an even, a natural, an integer, and so on. Yet I learn I got all those things if I learn I got a 2.

I’ve thought of other examples, though all of them deal with some single event occurring out of a set of infinite possible outcomes. E.g., pulling from the natural numbers: P(2|even) = P(2) = 0; but P(even|2) = 1 (rather than the P(even) = 1/2). So I imagine there’s something I’m naive about in the domain of infinite possible outcomes.

What am I missing?

$\endgroup$
2
  • 1
    $\begingroup$ Some of the independence formulas require the probabilities to be nonzero. $\endgroup$
    – Michael Burr
    Jun 8, 2018 at 23:35
  • $\begingroup$ Suppose the support for the distribution was limited to a set of numbers. You haven't "learned" you have a 2, you are finding how often you get a 2 given you know you picked a prime. i.e. out of the whole support, how many 2s divided by how many primes as seen in the answer below. $\endgroup$
    – Tony Hellmuth
    Jun 8, 2018 at 23:50

4 Answers 4

Reset to default
12
$\begingroup$

From the definition of conditional probability, we have that $$P(A\mid B)P(B) = P(A\cap B) = P(B\mid A)P(A).$$ Rearranging a bit, we see that $$ \frac{P(A\mid B)}{P(A)} = \frac{P(B\mid A)}{P(B)}. $$ $A$ is independent from $B$ iff the left-hand side is equal to $1$, and $B$ is independent from $A$ iff the right-hand side is equal to $1$. They're equal, so independence is symmetric.

$\endgroup$
4
  • $\begingroup$ But the question is really about conditioning on events of probability $0.$ $\qquad$ $\endgroup$
    – Michael Hardy
    Jun 9, 2018 at 0:04
  • 6
    $\begingroup$ @MichaelHardy I have to admit I struggled to understand what was being asked, but thought that having an answer to the question in the title would be a positive contribution. $\endgroup$
    – Jalex Stark
    Jun 9, 2018 at 0:07
  • $\begingroup$ Should be $P(A\cap B)$ instead of $P(A\wedge B)$ since those are not boolean variables but events (=sets). $\endgroup$
    – fabian
    Jun 9, 2018 at 7:29
  • 1
    $\begingroup$ BTW: notice that this result does not apply to expectations ("predictability"): $E(A|B)=E(A)$ does not imply $E(B|A)=E(B)$ . math.stackexchange.com/questions/626958/… $\endgroup$
    – leonbloy
    Jun 9, 2018 at 16:36
11
$\begingroup$

First problem: Choose a number from all real numbers, uniformly distributed? There is no such distribution on $\Bbb R$.

Second problem: If you have any probability space $(\Omega,\mathcal F,P)$ and an event $A\in\mathcal F$, then a probability space $(A,\mathcal F|_A,P')$ is only induced (via $P'(S)=\frac{P(S)}{P(A)}$) if $P(A)>0$.

$\endgroup$
0
7
$\begingroup$

  1. The symmetry of independence is manifest in its definition $P(A\cap B) = P(A)P(B)$ which makes no reference to conditional probability.

  2. Conditional probability is problematic when the conditioning event has probability zero, as evidenced by the division by zero that would occur in the definition $P(A\mid B) = \frac{P(A\cap B)}{P(B)}.$ For it to have meaning, we must be in a situation where a limit of $B$ approaching a null event is understood.

  3. Your example is problematic in other ways. There is no uniform distribution on the real line, nor is there a uniform distribution on the prime numbers. It does not make sense to say the probability that a prime is even is zero. There is a related concept of density, but there is a good reason why density is not regarded as the same thing as probability.
  4. A less problematic variant of your question would be to look at something like a standard normal $X$ and then consider the events $X>0$ and $X=2.$ It does seem a little strange that we have independence since $$P(X>0,X=2) = P(X=2) = 0 =P(X>0)P(X=2) $$ whereas clearly $X=2$ implies $X>0,$ and we want to write something like $$P(X>0\mid X=2)=1$$ even though it's undefined as a conditional probability (and indeed, we can write something like it... see the caveat to point (2) above regarding there being a well-defined limit.) This is probably best viewed as a counterintuitive fact about independence when events have probability zero or one. An event with probability zero or one is always independent of any other event. As a particular extreme case, note this means an event with probability zero or one is independent of itself!
$\endgroup$
6
  • $\begingroup$ Thanks for the thoughtful response. Regarding 3: several folks have pointed out that there's no uniform distribution on the real line, and you've pointed out that this is also true for the prime numbers. By "uniform distribution," I meant that any given number had the same probability of being selected. Did I use the wrong term? And, if so, is there a better term to use? Or, rather, is there something problematic about saying that all elements of an (uncountably or countably) infinite set could have the same probability of being selected? $\endgroup$
    – budford
    Jul 1, 2018 at 2:40
  • $\begingroup$ @jandawl It is problematic to say that for a countably infinite set. That probability would have to be zero, since otherwise all the probabilities can't add up to one since (roughly speaking) infinity times any nonzero number is infinity. But if the probabilities are all zero, then they can't all add up to one either. Thus there is no uniform distribution on any countably infinite set. Your definition of uniform distribution is correct for countably infinite sets (and thus this comment, which uses your definition, applies), but for uncountably infinite sets it's more complicated. (ctd) $\endgroup$
    – spaceisdarkgreen
    Jul 1, 2018 at 2:57
  • $\begingroup$ In all continuous distributions on the real line, every number has the same probability... zero, so are all uniform by your definition (which is not right here). There are tons of real numbers, and if we gave them all nonzero probability that wouldn't work by the above argument, so instead we assign probability for the number to lie in small intervals (called a density function), but any individual number has probability zero (our argument above for why this can't happen fails for uncountable sets, because probability distributions are only countably additive, not uncountably). (ctd) $\endgroup$
    – spaceisdarkgreen
    Jul 1, 2018 at 3:04
  • $\begingroup$ So for uncountable subsets of the reals, we can have uniform distributions if instead we use the definition that the density function be constant, rather than the probability of a given number be constant. There is a uniform distribution on an interval $[a,b]$... just let the probability of lying in an interval of size $dx$ be proportional to $dx.$ $\int_a^b dx =(b-a)$ so we can let the density be $\frac{1}{b-a}dx$). This won't work on the whole real line, since $\int_{-\infty}^\infty dx = \infty,$ so our probability of being anywhere can't be one. $\endgroup$
    – spaceisdarkgreen
    Jul 1, 2018 at 3:20
  • $\begingroup$ So by my first comment we don't have a uniform distribution on the primes (since they're a countably infinite set) and by my third we don't have a uniform distribution on the reals. $\endgroup$
    – spaceisdarkgreen
    Jul 1, 2018 at 3:21
4
$\begingroup$

What a great question!

OK, OK, so you're facing a bit of flak for the setup... but in this case we can somewhat make it rigorous!

Consider a geometric distribution $\text{Pr}(k) = p~(1-p)^{k-1}$ over the natural numbers $\mathbb N^+ = \{1,2,3,\dots\}$. We can build a table of the probabilities that you are interested in for the various $p$ values.

These numbers were obtained by Haskell (if you want to replicate, cabal install primes followed by declaring smallPrimes = takeWhile (< 10000000) primes :: [Int] and then declaring prprime p = sum . reverse . map (\k -> p * (1-p)^(k-1)) $ smallPrimes) so I'd expect there's going to be some round-off errors, but:

   p    | Pr(2)         | Pr(prime)     | Pr(2 & prime) | Pr(2 | prime) | Pr(prime | 2)
--------+---------------+---------------+---------------+---------------+---------------
 0.5    | 0.25          | 0.41468250985 | 0.25          | 0.60287085677 | 1.0
 0.333  | 0.222111      | 0.47466583522 | 0.222111      | 0.46793129718 | 1.0
 0.1    | 0.09          | 0.41253315631 | 0.09          | 0.21816428237 | 1.0
 0.03   | 0.0291        | 0.30889192395 | 0.0291        | 0.09420770743 | 1.0
 0.01   | 0.0099        | 0.24179453736 | 0.0099        | 0.04094385303 | 1.0
 0.003  | 0.002991      | 0.19183171475 | 0.002991      | 0.01559179098 | 1.0
 0.001  | 0.000999      | 0.16020745005 | 0.000999      | 0.00623566507 | 1.0
 0.0003 | 0.00029991    | 0.13500470068 | 0.00029991    | 0.00222147820 | 1.0
 0.0001 | 0.00009999    | 0.11779050920 | 0.00009999    | 0.00084887994 | 1.0

As you go to lower and lower $p$ the distribution becomes flatter and flatter among the first several numbers. We can see some immediate trends: including as you surmise that $\text{Pr}(k\text{ prime})$ is decreasing slowly as it should be, and that $\text{Pr}(k=2|k\text{ prime})$ is decreasing quickly like it should be. What you also notice is the fact that "2 is prime" is reflected directly in the rightmost column and it causes $\text{Pr}(k=2 \cap k \text{ prime}) = \text{Pr}(k = 2)$ directly, which means that unless $\text{Pr}(k\text{ prime}) = 1$ you are totally right to guess that these are dependent variables upon each other.

What you can see from this table is also where you're going wrong in the argument. When you say that $\text{Pr}(k = 2) \approx \text{Pr}(k=2|k\text{ prime})$ that is not borne out by the above data and in fact that $\approx$ sign is consistently off by a factor of $1/\text{Pr}(k\text{ prime}),$ so it starts off wrong by a factor of 2.4 or so and then this just gets worse until by the bottom of this table it is wrong by almost a factor of 10, and just as we know this probability is getting smaller and smaller, we know that this approximation is getting wronger and wronger. In general you would get a higher probability of finding $k=2$ if you knew that you happened to land upon the primes, and they are not truly independent events for any real assignment of probabilities that tries to remain relatively even for the first $N$ numbers before tapering off.

$\endgroup$
8
  • $\begingroup$ Hi, thanks for this! Very interesting. I'm confused, though. My understanding is that if you have an infinite set, the probability of pulling out any particular element of that set is 0. Especially if the pulling process is such that all elements have the same probability of being pulled. So, the probability of pulling a 2 from the set of all evens would be 0, right? Same for pulling a 2 from all primes? Same for pulling a prime from all integers (or at least the reals)? I see that, in your chart, you're tending the probability of 2 towards 0, but what happens when it just is 0? $\endgroup$
    – budford
    Jun 30, 2018 at 16:39
  • $\begingroup$ @jandawl That understanding is not 100% right and the appropriate mathematical machinery may be overkill, but it goes like this: You have a set of “points” and a probability function which maps subsets of points to numbers between 0 and 1 with certain rules like $P(A\cup B)=P(A)+P(B)-P(A\cap B)$ and such. This works for sets of infinite points and does not place any restriction on say $P(\{x\})=0$. Like it is very common for certain sorts of infinite, like if your set is $\mathbb R$ then the uniform distribution on $(0,1)$ has this property, but many other probability functions don't. $\endgroup$
    – CR Drost
    Jun 30, 2018 at 21:13
  • $\begingroup$ Thanks for your response! I'm afraid I don't follow. Perhaps I can break this down. (1) If you choose a random number from ℝ, the probability is 0 for any given number being chosen. Right? (2) The probability that a random number chosen from ℝ has a probability of zero of being prime, as the primes make up "precisely 0%" of the reals. Are (1) and (2) correct? If not, what are the correct answers? (3) You mention uniform distribution on (0,1). Does that go against what many have said here, i.e., "there is no such distribution on ℝ"? Or is it Ok because it's an open interval? $\endgroup$
    – budford
    Jun 30, 2018 at 22:05
  • $\begingroup$ PS: I'm also extremely curious to know, then: (4) What is the probability of pulling a 2 from the set of all even numbers (and/or from the natural numbers)? (5) What is the probability of pulling a prime from the set of all integers (and/or from the natural numbers). Since (4) and (5) are apparently non-0 probabilities, what are those probabilities? $\endgroup$
    – budford
    Jun 30, 2018 at 22:12
  • $\begingroup$ It's great to see someone who is committed to getting this stuff down right! You are missing the idea that there are many distributions. It is easiest to see this in the discrete case. I can generate a number 1-6 by rolling a fair die. I can also generate these by flipping 5 coins, counting the number of heads, and adding 1 to it. I can also generate this by rolling two fair dice, then taking the maximum of the two rolls. What is the probability of rolling a 6? It depends which $P()$ I use: in the first case $1/6$, in the second $1/32$, in the third $11/36$. I need $P$ before I can answer. $\endgroup$
    – CR Drost
    Jul 1, 2018 at 1:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .