Let $f_X(x)$ and $g_{Y}(x)$ be probability mass functions of discrete random variables X and Y. Mike selects a random variable (he chooses $X$ with probability $1/2$ or $Y$ with probability $1/2$), then he generates a sample of it and gives it to us. Let $a$ be the number that we get. We don't know which random variable was selected. Based on the observation $a$, find the probability that he has selected $X$.

Let ${A}$ be the event that $a$ is observed. To answer this question, we have to calculate: \begin{align} {P}(a \text{ is a sample of } X|A)=\frac{P(A \cap \{X\text{ selected}\})}{P(A)}&=\frac{P(A | X\text{ selected})P(X\text{ selected})}{P(A|X \text{ selected })0.5+P(Y \text{ selected })0.5}\\&=\frac{f_X(a) 0.5}{P(A|X \text{ selected })0.5+P(Y \text{ selected })0.5}\\ &=\frac{f_X( a)}{f_X( a)+g_Y( a)}\\ \end{align}

How can we extend this to the continuous random variables?

For each probability density function (pdf) the probability of observing $a$ is zero. So, we cannot use the above math to calculate the probability we need. But, intuitively, we can have examples of $X$s and $Y$s such that their support includes $a$, but one of them is more centered at $a$, so, it is more probable that it is generated from the one centered at $a$. How can we measure how much it is probable that $X$ generated $a$?


Consider a small interval around $a$, i.e., the observed value to be in the interval $[a -\varepsilon, a+\varepsilon]$ and then take the limit $\varepsilon \to 0$ when evaluating the ratio.

Then the ratio becomes \begin{equation} \mathrm{lim}_{\varepsilon \to 0} \frac{\int^{a+\varepsilon}_{a-\varepsilon}f_X (x) dx}{\int^{a+\varepsilon}_{a-\varepsilon}f_X (x) dx \, + \int^{a+\varepsilon}_{a-\varepsilon}g_Y (y) dy } = \frac{f_X(a)}{f_X(a)+g_Y(a)} \end{equation}

For an illustration, let us consider $X \sim \mathcal{N}$(0,1) and $Y \sim \mathcal{N}(1,1)$ and take the observed value to be $a$.

\begin{align} \mathrm{lim}_{\varepsilon \to 0} \frac{\int^{a+\varepsilon}_{a-\varepsilon}f_X (x) dx}{\int^{a+\varepsilon}_{a-\varepsilon}f_X (x) dx \, + \int^{a+\varepsilon}_{a-\varepsilon}g_Y (y) dy } &= \mathrm{lim}_{\varepsilon \to 0}\frac{ \int^{a+\varepsilon}_{a-\varepsilon} \mathrm{e}^{-x^2/2} dx}{\int^{a+\varepsilon}_{a-\varepsilon} \mathrm{e}^{-x^2/2} dx \, + \, \int^{a+\varepsilon}_{a-\varepsilon} \mathrm{e}^{-(y-1)^2/2} dy} \end{align}
Using the Liebniz integral rule of differentiating under the integral sign to calculate the derivative for applying L'Hopital's rule, \begin{equation} \mathrm{lim}_{\varepsilon \to 0} \frac{e^{-(a+\varepsilon)^2/2}+e^{-(a-\varepsilon)^2/2} }{e^{-(a+\varepsilon)^2/2}+e^{-(a-\varepsilon)^2/2} + e^{-(a+\varepsilon-1)^2/2}+e^{-(a-\varepsilon-1)^2/2} } = \frac{e^{-(a)^2/2} }{e^{-(a)^2/2} + e^{-(a-1)^2/2} } \end{equation}

So, if the observed value is $0$, the probability that it came from the distribution $X$ $\approx .6225$, which is higher, like we would expect.

| cite | improve this answer | |
  • $\begingroup$ Thanks. Does it mean we are only replacing the pmfs with pdfs? $\endgroup$ – Susan_Math123 Jun 13 '18 at 15:51
  • $\begingroup$ Because if we take the derivatives, we obtain pdfs $\endgroup$ – Susan_Math123 Jun 13 '18 at 15:52
  • $\begingroup$ replace the each integral in your first equation with diffrences of CDFs, and continue , you will see you get pdfs after differentiating $\endgroup$ – Susan_Math123 Jun 13 '18 at 15:53
  • $\begingroup$ Yes, we are only replacing the pmfs with pdfs. $\endgroup$ – Mohit Jun 13 '18 at 18:25
  • $\begingroup$ I'm a bit confused. If we're using the Liebniz integral rule of differentiation, shouldn't we have "-" signs for each pair of terms from the definite integrals? In that case, don't we get 0/0? $\endgroup$ – brian.keng Aug 1 '19 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.