7
$\begingroup$

Consider the model $X \sim N(\theta, 1)$ for $\theta \in \mathbb R$. Obtain the conditional distribution of $X$ given $X^2=t$ for $t\ge 0$.

Hint: This is a discrete distribution.

Current work: $f(X=x|X^2=t)=\displaystyle \frac{f(X=x, X^2=t)}{f(X^2=t)}=\frac{f(X=x, X^2=t)}{\frac{1}{2\sqrt{2\pi t}}(e^{(\sqrt t-\theta)^2/2}+e^{(\sqrt t + \theta)^2/2})}=\frac{f(X=\sqrt t)}{\frac{1}{2\sqrt{2\pi t}}(e^{(\sqrt t-\theta)^2/2}+e^{(\sqrt t + \theta)^2/2})}=\frac{\frac{1}{\sqrt {2\pi}}e^{-(\sqrt t-\theta)^2/2}}{\frac{1}{2\sqrt{2\pi t}}(e^{(\sqrt t-\theta)^2/2}+e^{(\sqrt t + \theta)^2/2})}$? But this can't be right because there's no $x$ in the answer. What does one do here?

$\endgroup$
6
  • 5
    $\begingroup$ Given $X^2=t$, the random variable $X$ must either be $\sqrt{t}$ or $-\sqrt{t}$; it remains to compute the probability of each outcome. $\endgroup$ – angryavian Jan 10 at 2:45
  • 1
    $\begingroup$ @angryavian $P(X=\sqrt{t})=P(X=-\sqrt{t})=0$ if $X\sim N(\theta,1)$ $\endgroup$ – Matthew Pilling Jan 10 at 4:14
  • 1
    $\begingroup$ It is clear that the density is discrete and exists only when $x = \sqrt t$ or $x = -\sqrt t$. In your second equality, $f(X = x , X^2 = t)$ is then either $X = \sqrt t$ or $X = - \sqrt t$ (you continued with just $x = \sqrt t$, which is false since it could be the negative as well). Also, there's no $x$ in the answer because $x$ is written in terms of $t$. Replace all the $\sqrt t$ by $x$ and you have $x$ in your answer. It is nearly correct, basically, but you just have to do the two possible values of $x$ separately. $\endgroup$ – Teresa Lisbon Jan 10 at 7:13
  • 4
    $\begingroup$ I have another issue : we usually don't condition on measure zero events, do we? So technically, we need to approximate the given event by events with probability approaching zero, and then see if the resulting conditional laws converge weakly to a random variable, which we regard as the conditional expectation. I am sure the formula continues to hold because weak convergence preserves it, but this is an important point. $\endgroup$ – Teresa Lisbon Jan 10 at 8:20
  • $\begingroup$ @MatthewPilling What I meant was that if we know a probability distribution (in this problem, the conditional distribution) is supported on two points, then we just need to compute the probability of those two points to specify the distribution. I didn't mean that these two probabilities were $P(X=\sqrt{t})$ and $P(X = -\sqrt{t})$. $\endgroup$ – angryavian Jan 10 at 16:48
6
$\begingroup$

Disclaimer: I'm not an expert, so please correct me if there are serious flaws in my argument.

Hand-waving/guessing

First observe that if $X^2=t$ then $X$ is either $\sqrt{t}$ or $-\sqrt{t}$, so this is a discrete distribution, and to specify the distribution you only need to compute $P(X=\sqrt{t} \mid X^2=t)$ and $P(X=-\sqrt{t} \mid X^2=t)$.

In analogy with the discrete case, you might rewrite $\{X^2 = t\} = \{X=\sqrt{t}\} \cup \{X = -\sqrt{t}\}$ and guess $$P(X=\sqrt{t} \mid X^2=t) = \frac{f_X(\sqrt{t})}{f_X(\sqrt{t}) + f_X(-\sqrt{t})}$$ $$P(X=-\sqrt{t} \mid X^2=t) = \frac{f_X(-\sqrt{t})}{f_X(\sqrt{t}) + f_X(-\sqrt{t})}$$

This would certainly be a probability distribution since the probabilities sum to one. However it is not clear why we can add/divide the densities in this way, since the density values are not really probabilities. The discussion below gives a little more background on why this ends up being valid. However, in my experience undergraduate courses are content to sweep this under the rug since this discussion would involve some rather technical measure theory concepts.

Before I turn to the more rigorous discussion, I want to comment that I noticed your denominator is the density of $f_{X^2}$ (which you correctly noted to be $f_{X^2}(t) = \frac{1}{2\sqrt{t}}(f_X(\sqrt{t}) + f_X(-\sqrt{t}))$. However, you used $f_X(\sqrt{t})$ as the numerator. Due to the change of variables, the $\frac{1}{2\sqrt{t}}$ scaling factor prevents your attempt from being a probability distribution (in that $\frac{f_X(\sqrt{t})}{f_{X^2}(t)} + \frac{f_X(-\sqrt{t})}{f_{X^2}(t)} \ne 1$). This is one reason why it can be dangerous to manipulate densities as you would with discrete probabilities, even if sometimes you do get the correct answer.

Verifying guess with slightly more technical stuff

Conditioning on events of measure zero is a bit of a thorny discussion. One framework for handling this situation is the notion of regular conditional probabilities. The definition translates to the following. The map $(t, A) \mapsto \nu(t, A)$ is the regular conditional probability of $X$ given $X^2$ if

  1. $\nu(t, \cdot)$ is a probability measure for each $t$
  2. $\nu(\cdot, A)$ is measurable for each measurable set $A$
  3. $P(\{X \in A\} \cap \{X^2 \in B\}) = \int_B \nu(t, A) f_{X^2}(t) \, dt, \qquad \text{for all measurable sets $A,B$}.$

(Note that the right-hand side of the third condition can be written as $E[\nu(X^2, A) \mathbf{1}_{X^2 \in B}]$.)

Then $\nu(t, A)$ is what we would usually denote by $P(X \in A \mid X^2 = t)$.

Let's try the guess we made above. That is, let's try $$\nu(t, A) = \frac{f_X(\sqrt{t})}{f_X(\sqrt{t}) + f_X(-\sqrt{t})} \mathbf{1}_A(\sqrt{t}) + \frac{f_X(-\sqrt{t})}{f_X(\sqrt{t}) + f_X(-\sqrt{t})}\mathbf{1}_A(-\sqrt{t}).$$ This satisfies the first two conditions. It remains to check the third condition.

Notice that for $t \ge 0$, we have $f_{X^2}(t) = \frac{d}{dt} P(X^2 \le t) = \frac{d}{dt} \int_{-\sqrt{t}}^\sqrt{t} f_X(s) \, ds = \frac{1}{2\sqrt{t}} (f_X(\sqrt{t}) + f_X(-\sqrt{t}))$. (This is in the denominator of your attempt.) Let $B_+ := B \cap [0, \infty)$. Then, \begin{align} &\int_{B} \nu(t, A) f_{X^2}(t) \, dt \\ &= \int_{B_+} \frac{1}{2\sqrt{t}} f_X(\sqrt{t}) \mathbf{1}_A(\sqrt{t}) \, dt + \int_{B_+} \frac{1}{2\sqrt{t}} f_X(-\sqrt{t}) \mathbf{1}_A(-\sqrt{t}) \, dt \\ &= \int_0^\infty f_X(u) \mathbf{1}_A(u) \mathbf{1}_{B_+}(u^2) \, du + \int_0^\infty f_X(-u) \mathbf{1}_A(-u) \mathbf{1}_{B_+}((-u)^2) \, du \\ &= \int_{-\infty}^\infty f_X(v) \mathbf{1}_A(v) \mathbf{1}_{B_+}(v^2) \, dv \\ &= P(\{X \in A\} \cap \{X^2 \in B\}). \end{align}

Alternate definition involving a limit of events with vanshing probability

Teresa Lisbon and Matthew Pilling consider an alternate way to define conditioning on a zero-probability event by approximating it with events with probability approaching zero. This is discussed on the same Wikipedia page. In our situation, it would be

$$P(X \in A \mid X^2 = t ) = \lim_{\epsilon \to 0} \frac{P(\{X \in A\} \cap \{t-\epsilon < X^2 < t + \epsilon\})}{P(t-\epsilon < X^2 < t + \epsilon)}.$$

I have not checked whether this alternate definition yields the same answer as above, but I want to note that this is not always considered a "good" way to define conditional probability. Indeed there is a warning on that Wikipedia section with a link to a discussion that contains a simple example for where this definition leads to an irregular conditional probability distribution.

Consider $U \sim \text{Uniform}(0,1)$. For any $\delta > 0$, we have \begin{align} &P(0.5-\delta < U < 0.5+\delta \mid U=0.5) \\ &= \lim_{\epsilon \to 0} \frac{P(\{0.5-\delta < U < 0.5+\delta\} \cap \{0.5-\epsilon < U < 0.5+\epsilon\})}{P(0.5-\epsilon < U < 0.5+\epsilon)} \\ &= 1\end{align} but $$P(U=0.5 \mid U=0.5) = \lim_{\epsilon \to 0} \frac{P(\{U=0.5\} \cap \{0.5-\epsilon < U < 0.5+\epsilon\})}{P(0.5-\epsilon < U < 0.5+\epsilon)} = 0.$$

Appendix

My understanding of conditional probabilities may be shaky so please correct any misunderstandings in my answer. I found the Wikipedia pages useful (linked above) as well as this answer by Stefan Hansen.

$\endgroup$
3
  • $\begingroup$ (+ 1000000) This is wonderful. $\endgroup$ – Matthew Pilling Jan 10 at 20:15
  • $\begingroup$ this is awesome! The next part of the problem actually asks to show the third condition, $\int \mathbb P(X \in I | X^2=t)f_{X^2}(t)dt=\mathbb P(X \in I)$ $\endgroup$ – Stacker Jan 11 at 4:25
  • $\begingroup$ Yes , this makes far more sense! +1 $\endgroup$ – Teresa Lisbon Jan 11 at 5:22
3
$\begingroup$

I'm wondering if this conditional distribution is even well$-$defined. Let $\epsilon_1,\epsilon_2$ be very small positive numbers and $t>0$ be arbitrary. Define $I_1=(-\sqrt{t}-\epsilon_1,-\sqrt{t}+\epsilon_1)$ and $I_2=(\sqrt{t}-\epsilon_2,\sqrt{t}+\epsilon_2)$. Assume $\epsilon_1, \epsilon_2$ are small enough to make $I_1,I_2$ disjoint. Notice how $$P\Big(X\in I_1 \Big|X\in I_1 \cup I_2\Big)=\frac{\int_{-\sqrt{t}-\epsilon_1}^{-\sqrt{t}+\epsilon_1}f_{X}(x)dx}{\int_{-\sqrt{t}-\epsilon_1}^{-\sqrt{t}+\epsilon_1}f_{X}(x)dx+\int_{\sqrt{t}-\epsilon_2}^{\sqrt{t}+\epsilon_2}f_{X}(x)dx}\approx \frac{\epsilon_1 f_{X}\big(-\sqrt{t}\big)}{\epsilon_1 f_{X}\big(-\sqrt{t}\big)+\epsilon_2 f_{X}\big(\sqrt{t}\big)}$$ One would expect that the above expression approaches $P\Big(X=-\sqrt{t}\Big|X^2=t\Big)$ as $(\epsilon_1,\epsilon_2)\rightarrow 0$ but such a limit doesn't exist. I think the most natural conditional pmf would be $$P(X=s|X\in S)=\frac{f_{X}(s)}{\sum_{x\in S}f_{X}(x)}$$ where $X\sim f_{X}$ is any continuous random variable supported on $\mathbb{R}$, $S\subseteq \mathbb{R}$ is any finite set, and $s\in S$ is fixed, but you could repeat a similar argument above and see something weird happening. FYI: This is not intended to be an answer, but my response is too long for a comment.

$\endgroup$
1
  • $\begingroup$ I'm not an expert, but I think this is not the conventional way to define conditioning on a zero-probability event; see the end of my answer. $\endgroup$ – angryavian Jan 10 at 19:36
2
$\begingroup$

MAJOR EDIT

Better check @angryavian's answer.

$\Pr\{X=x|X^2=t\}$

$\;\;\;=\Pr\{X=x|X>0, X^2=t\}\Pr\{X>0|X^2=t\}$

$\;\;\;\;\;\; +\Pr\{X=x|X<0, X^2=t\}\Pr\{X<0|X^2=t\}$

Now:

$\Pr\{X=x|X>0, X^2=t\}=1$, if $x=\sqrt{t}$, and zero otherwise,

also $\Pr\{X=x|X<0, X^2=t\}=1$, if $x=-\sqrt{t}$, and zero otherwise.

And:

$\Pr\{X>0|X^2=t\} = \dfrac{\Pr\{X^2\approx t,X>0\}}{\Pr\{X^2\approx t\}}$

$\;\;\;\; =\dfrac{f_X(\sqrt{t})\left|d(x^2)|_{x=\sqrt{t}}\right|}{f_X(\sqrt{t})\left|d(x^2)|_{x=\sqrt{t}}\right|+f_X(-\sqrt{t})\left|d((-x)^2)|_{x=-\sqrt{t}}\right|}$

$\;\;\;\;= \dfrac{\frac{1}{\sqrt{2\pi}}\exp\{-0.5(\sqrt{t}-\theta)^2\}}{\frac{1}{\sqrt{2\pi}}\exp\{-0.5(\sqrt{t}-\theta)^2\}+\frac{1}{\sqrt{2\pi}}\exp\{-0.5(-\sqrt{t}-\theta)^2\}} = \dfrac{e^{\theta\sqrt{t}}}{e^{\theta\sqrt{t}}+e^{-\theta\sqrt{t}}}$

In a similar way:

$\Pr\{X<0|X^2=t\} = \dfrac{e^{-\theta\sqrt{t}}}{e^{\theta\sqrt{t}}+e^{-\theta\sqrt{t}}}$

So,

$\Pr\{X=x|X^2=t\} =\dfrac{e^{-\theta\sqrt{t}}}{e^{\theta\sqrt{t}}+e^{-\theta\sqrt{t}}}\delta(x+\sqrt{t})+\dfrac{e^{\theta\sqrt{t}}}{e^{\theta\sqrt{t}}+e^{-\theta\sqrt{t}}}\delta(x-\sqrt{t}) $

$\endgroup$
6
  • $\begingroup$ The first equality is false. In general, $P(A \mid B) = P(A \mid B,C) P(C\mid B) + P(A \mid B, \lnot C) P(\lnot C \mid B)$. $\endgroup$ – angryavian Jan 10 at 16:49
  • $\begingroup$ I just dropped the trivial: $\Pr\{X>0|X^2=t\}=\Pr\{X>0\}$. $\endgroup$ – Arash Jan 10 at 19:56
  • $\begingroup$ That isn't true when $\theta \ne 0$. $\endgroup$ – angryavian Jan 10 at 20:01
  • $\begingroup$ Actually I suspect that it is correct in general, i.e., regardless of dist. $\endgroup$ – Arash Jan 10 at 20:06
  • $\begingroup$ If $X$ takes values $-2, -1, 1, 2$ with probabilities $1/6, 1/3, 1/6, 1/3$ respectively, then $P(X>0)=1/2$ but $P(X>0 \mid X^2=1) = \frac{1/6}{1/6 + 1/3} = 1/3$. $\endgroup$ – angryavian Jan 10 at 20:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.