Consider the following experiment:

We generate "green" points and "blue" points in $\mathbf{R}$ using two different normal distributions as follows:

  • 1000 green points are sampled from a $N(-1, 1)$ distribution
  • 1000 blue poiints are sampled from a $N(1,1)$ distribution

Now I hide the colors of the points, point to one of them at random and ask you: What is the probability that this point is green?

I think we want: $$P(g | X=x) = \frac{P(X=x | g)P(g)}{P(X=x)}.$$

Now I think $P(g)=\frac{1}{2}$, since green and blue are equally likely, and $P(X=x) = \frac{1}{2000}$ since I am showing you one of 2000 points with equal probability. What throws me off is the $P(X=x | g)$. This is the probability that $X=x$ when $X$ is distributed as $N(-1,1)$. Isn't this simply $0$ since $X$ is a continuous random variable? Am I way off here?

  • $\begingroup$ Do we get to see the locations (though not the colors) of all $2000$ points and have to answer the question about one of them, or are you going to show just one point $3.213$, say, and ask "What's the probability that this is a green point?" $\endgroup$ – Dilip Sarwate Aug 25 '15 at 12:44

You have the right idea but the wrong Bayes formula.

You need to replace $P(X=x|g)$, which is indeed $0$, by $f_g(x)$ where $f_g$ is the pdf of $N(-1,1)$.

Similarly, $P(X=x)=0$ but you replace it by $f_g(x)P(g)+f_b(x)P(b)$. This is called a gaussian mixture.


The idea behind is OK but the following expression is wrong:

$$P(g | X=x) = \frac{P(X=x | g)P(g)}{P(X=x)}.$$

Here is the right approach. Say, first, that the a black random point (denoted by $X$) lies in the interval $[x,x+\Delta x]$. Then we have

$$P(g\mid x\le X \le x+\Delta x)=\frac{P(\{x\le X \le x+\Delta x\} \cap g)}{P(x\le X \le x+\Delta x)}=\frac{P(x\le X \le x+\Delta x \mid g)\frac12}{P(x\le X \le x+\Delta x)}.$$

Now, if $\Delta x$ is small then

  1. $P(x\le X \le x+\Delta x \mid g)\approx f_g(x)\Delta x$.
  2. $P(x\le X \le x+\Delta x)=\frac12(P(x\le X \le x+\Delta x\mid g)+P(x\le X \le x+\Delta x\mid b))\approx \frac12(f_g(x)\Delta x+f_b(x)\Delta x) $

With this

$$P(g\mid x\le X \le x+\Delta x)\approx\frac{\frac12f_g(x)\Delta x}{\frac12(f_g(x)\Delta x+f_b(x)\Delta x)}.$$

(For the sake of simplicity, I've assumed that $P(g)=P(b)=\frac12.$)

Apparently $\Delta x$ does not have a real role in this argumentation. It is only a slight technical problem to change $\approx$ to a $\lim\limits_{\Delta x \to 0} $ and an $=$.

So, we have

$$P(g\mid X=x)=\frac{f_g(x)}{f_g(x)+f_b(x)}.$$

Or, in general

$$P(g\mid X=x)=\frac{P(g)f_g(x)}{f_g(x)P(g)+f_b(x)P(b)}.$$

  • $\begingroup$ In your last two equations, the numerator should be the joint distribution of $x$ and $g$, but isn't $f_g(x)$ the distribution of $x$ given $g$? $\endgroup$ – Hilemonstoer Aug 25 '15 at 13:19
  • $\begingroup$ @Hilemonstoer: I edited my answer. The problem was that I forgot about $\frac12$ and left out one step: $$P(g\mid x\le X \le x+\Delta x)=\color{red}{\frac{P(x\le X \le x+\Delta x \cap g)}{P(x\le X \le x+\Delta x)}}=\frac{P(x\le X \le x+\Delta x \mid g)\frac12}{P(x\le X \le x+\Delta x)}$$ $\endgroup$ – zoli Aug 25 '15 at 13:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.