4
$\begingroup$

$X,Y\sim N(0,1)$ and are independent, consider $X+Y$ and $X-Y$.

I can see why $X+Y$ and $X-Y$ are independent based on the fact that their joint distribution is equal to the product of their marginal distributions. Just, I'm having trouble understanding intuitively why this is so.

This is how I see it : When you look at $X+Y=u$, the set $\{(x,u-x)|x\in\mathbb{R}\}$ is the list of possibilities for $X$ and $Y$.

And intuitively, I understand independence of two random variables $A$ and $B$ as, the probability of the event $A=a$ being completely unaffected by the event $B=b$ happening.

But when you look at $X+Y=u$ given that $X-Y=v$, the set of possibilities has only one value $(\frac{u+v}{2},\frac{u-v}{2})$.

So, $\mathbb{P}(X+Y=u|X-Y=v)\neq \mathbb{P}(X+Y=u)$.

Doesn't this mean that $X+Y$ is affected by the occurrance of $X-Y$? So, they would have to be dependent? I'm sorry if this comes off as really stupid, it has been driving me crazy, even though I am sure that they are independent, it just doesn't feel right.

Thank you.

$\endgroup$
2
$\begingroup$

(1) The short, short answer is that it is wrong to say

$$\mathbb{P}(X+Y=u|X-Y=v)\neq \mathbb{P}(X+Y=u)\,\,\,\,\,\,\text{(this is wrong)}$$

because in fact, both sides $=0$, as these are continuous variables.

(2) The longer answer... Well first of all, the proper way to decide independence is to look at the joint PDF of $U = X+Y$ and $V=X-Y$, as you have already done. This is equivalent to checking:

$$f_U(U = u) \overset{?}= f_{U|V}(U = u \mid V = v) \equiv \frac{f_{U,V}(U = u \cap V = v)}{f_V(V = v)}$$

where you will find that both sides are non-zero and indeed equal.

(3) However, I wonder if your confusion comes from a more basic misunderstanding. It is of course true that $(U,V) = (u,v)$ defines exactly a single point in $(X,Y)$ space. However this does not automatically imply the conditional (prob or density) is $<$ the unconditional. After all, remember that all conditional prob (or density) are ratios. So if the numerator is very small but the denominator is proportionally small, then the ratio is unchanged and the conditional prob (or density) equals the unconditional version.

In your example, the unconditional asks for hitting a certain line $X+Y = u$ within the entire $2$-D $(X,Y)$ plane, while the conditional asks for hitting a point within a specific line $X-Y = v$. As mentioned, both probabilities are zero, but as you verified, both densities are non-zero and equal.

(4) Finally, you might like to know that multivariate Gaussians are the only variables with this property. So that might explain why your gut just keeps telling you that $X+Y, X-Y$ "cannot possibly be independent" when $X,Y$ are independent. :) I was confused about this in the recent past -- see this for a brief further discussion.

$\endgroup$
6
  • $\begingroup$ Thank you! That does make a lot of sense. Also, how would you explain the independence of two continuous random variables intuitively? $\endgroup$ – notacompactspace Sep 18 '20 at 16:53
  • 1
    $\begingroup$ @notacompactspace - You're welcome, and also Welcome to MSE! :) The other answer by John Dawkins gave more details on my point (4) -- multivariate Gaussians are the only variables which are rotational invariant like this. $\endgroup$ – antkam Sep 18 '20 at 16:54
  • 1
    $\begingroup$ Independence of continuous variables are defined by the PDF formula, but if you must map them into probabilities, then independence means for any subsets $A, B$ of the real line (including intervals of the form $[u, u+du]$) we must have $P(U \in A, V \in B) = P(U \in A) P(V \in B)$. $\endgroup$ – antkam Sep 18 '20 at 16:57
  • 1
    $\begingroup$ @antkam Why don't we define independence with that last equation, rather than via PDFs? $\endgroup$ – J.G. Sep 18 '20 at 17:05
  • 1
    $\begingroup$ OK, maybe I misspoke earlier. I wouldn't be surprised if the equation with the subsets is the actual definition. It is also not entirely trivial to show the two are equivalent -- it probably requires more Measure Theory than I know. :) (E.g. the subsets themselves would need to be measurable.) $\endgroup$ – antkam Sep 18 '20 at 17:18
2
$\begingroup$

To understand a very intuitive brainstorming let's start with $X,Y$ iid $N(\theta;1)$ distribution.

You will probabily know that $X+Y$ is a "complete sufficient statistic" for $\theta$ while $X-Y\sim N(0;2)$ is independent of $\theta$ so it is "ancillary"

This is that $X+Y$ contains all the information about $\theta$ while $X-Y$ has no useful information...its distribution does not depend anymore from $\theta$

So they are independent


This intuitive brainstorming is, in poor words, Basu's Theorem

$\endgroup$
1
$\begingroup$

Intuitively, it's because the joint density of $X$ and $Y$ is rotation invariant, and the transformation from $(X,Y)$ to $((X+Y)/\sqrt{2},(X-Y)/\sqrt{2})$ is a rotation. Therefore $(X+Y,X-Y)$ has the same distribution as $(\sqrt{2}X, \sqrt{2}Y)$, and the random variables in this latter pair are independent.

$\endgroup$
1
$\begingroup$

Let $X$ and $Y$ be two random variables, with finite second moment. Consider the variables $Z_1=X-Y$ and $Z_2=X+Y$.

Their covariance is

$$\rm{Cov}(Z_1, Z_2) = E[(X-Y)(X+Y)] - E(X-Y)E(X+Y) = {\rm Var}(X) - {\rm Var}(Y).$$

So

$${\rm Var}(X) = {\rm Var}(Y) \implies \rm{Cov}(Z_1, Z_2) = 0,\;\;\; {\rm Var}(X) \neq {\rm Var}(Y) \implies \rm{Cov}(Z_1, Z_2) \neq 0.$$

So a necessary condition for independence of $Z_1$ and $Z_2$ is that ${\rm Var}(X) = {\rm Var}(Y)$. No matter what the marginal and joint distributions are of the variables involved, if the variances of the $X$ and $Y$ variables are not equal, the independence result cannot hold.

Given this, the second required condition for independence of $Z_1, Z_2$ is that their joint distribution is such that zero covariance implies independence. There are many such distribution families, not just the Normal. For example, if the joint distribution is of the Farlie-Gumbel-Morgenstern type.

PS: Now the interesting question becomes: assume that $X$ and $Y$ have no moments. Under which conditions $Z_1$ and $Z_2$ will be independent?

PS2: The above result does not make nor uses the assumption that $X,Y$ are independent random variables.

$\endgroup$
4
  • $\begingroup$ Yes, I understand this. I understand why with respect to definition of independence X+Y and X-Y are independent. I am only confused about the intuitive aspect, because it seems to me that the occurrence of X+Y affects the occurrence of X-Y. $\endgroup$ – notacompactspace Sep 19 '20 at 6:16
  • 1
    $\begingroup$ @notacompactspace I have generalized the answer to show the general conditions under which this property holds. This has nothing to do with the variables being Normal, so I don't know how much intuition can one inject into this. $\endgroup$ – Alecos Papadopoulos Sep 19 '20 at 9:24
  • $\begingroup$ I am totally out of my depth and have never heard of the Farlie-Gumbel-Morgenstern family. I was mainly quoting an answer to one of my older questions, which in turn quotes this paper, whose abstract (if I understand correctly) seems to say Gaussians are the only possibility. Is it possible that your Farlie-Gumbel-Morgenstern family cannot be constructed as $X+Y, X-Y$ for some independent $X,Y$? $\endgroup$ – antkam Sep 19 '20 at 20:33
  • $\begingroup$ @antkam These results use the assumption that $X, Y$ are independent. What I present above is more general. I included this remark in the post. $\endgroup$ – Alecos Papadopoulos Sep 19 '20 at 23:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.