# Independence of Random Variables and Joint Random Variables

I have an intuitive understanding of Joint Random Variables and Independence but I want to make sure my mathematical understanding on the topics are sound.

Let $$(\Omega_1,\mathcal{F}_1,P_1)$$ , $$(\Omega_2,\mathcal{F}_2,P_2)$$ be two probability spaces. Define

$$X:\Omega_1\longrightarrow \mathcal{R}$$ and $$Y:\Omega_2\longrightarrow \mathcal{R}$$ be two random variables.

We can define the probability product space $$(\Omega_1\times \Omega_2, \mathcal{F_1\times F_2}, P_1\times P_2)$$ for the joint random variable $$(X,Y)$$.

Then $$P_1\times P_2(X\in C, Y\in D)=P_1(X\in C)P_2(Y\in D)$$ by definition of the product measure no independence needed here.
The only way I can see the definition of independence holds if both $$X$$ and $$Y$$ are defined on the same probability space say $$(\Omega,\mathcal{F},P)$$ and we define $$P(X\in C, Y\in D)=P(X\in C\cap Y\in D)=P({\omega\in \Omega:X(\omega)\in C ,Y(\omega)\in D})$$.

Independence holds only if $$P(X\in C\cap Y\in D)=P({\omega\in \Omega:X(\omega)\in C ,Y(\omega)\in D})=P(\omega\in \Omega:X(\omega)\in C)P(\omega\in \Omega:Y(\omega)\in D)$$

That is Independence can only be defined when the joint random variables are defined on the same probability space. Is my understanding correct? If I am wrong can someone help clarify. On a side note when we define two random variables from the same population but have different parameters that is $$X\sim N(\mu_1,\sigma_1)$$ $$Y\sim N(\mu_2,\sigma_2)$$

while they can be defined on the same measurable space are they defined on different probability spaces so how can independence hold if not on a product probability measure?

• Indeed independence of $X, Y$ only makes sense when $X, Y$ are on the same probability space. Rule of thumb: If you have a problem that uses random variables $X,Y,Z$, unless otherwise stated, those are assumed to be on the same space. – Michael May 15 '20 at 18:43
• @Michael Then when they say that $X$ and $Y$ are independent in the case of the normals above are they not defined on different probability spaces or is it that the probability measure $P$ induces two different measures on X and Y , that is $P_X\sim N(\mu_1, \sigma_1)$ and $P_y\sim N(\mu_2, \sigma_2)$. – Noe Vidales May 15 '20 at 18:47
• Whenever someone says $X$ and $Y$ are independent, it always means $X, Y$ are on the same sample space $\Omega$, so $X(\omega)$ and $Y(\omega)$ are determined by the same outcome $\omega \in \Omega$. Rule of thumb: Do not worry about problems that involve two or more different spaces, most likely you will live your life and never see such problems. – Michael May 15 '20 at 18:49
• You can build a sample space from previous sets, say, $\Omega = \Omega_1 \times \Omega_2$, where the outcomes are 2-dimensional $\omega = (\omega_1, \omega_2)$, then proceed to solve a problem on the single space $\Omega$. Then all random variables have the form $X(\omega) = X(\omega_1, \omega_2)$ and $Y(\omega) = Y(\omega_1, \omega_2)$. If those variables only depend on the first coordinate $\omega_1$, or the second $\omega_2$, so be it. But the outcome is $\omega=(\omega_1, \omega_2)$ and the outcome determines the values of all random variables in your problem. – Michael May 15 '20 at 19:01
• You also cannot define a probability measure as you did. You would first need to define the smallest sigma algebra that contains the set product of your two earlier algebras, and so on. Then you would need fundamental results on extending functions defined over a portion of a sigma algebra to the full sigma algebra. You may want to look up pi-systems and so on for these foundational concepts. – Michael May 18 '20 at 12:37

### Originally we have two different spaces

Let $$(\Omega_1, F_1, P_1)$$ and $$(\Omega_2, F_2, P_2)$$ be two probability spaces. That is, $$\Omega_1$$ and $$\Omega_2$$ are nonempty sets, $$F_1$$ is a sigma-algebra on $$\Omega_1$$, $$F_2$$ is a sigma-algebra on $$\Omega_2$$, and $$P_1$$ and $$P_2$$ are functions \begin{align*} P_1: F_1 \rightarrow\mathbb{R}\\ P_2:F_2 \rightarrow \mathbb{R} \end{align*} that satisfy the 3 probability axioms with respect to $$(\Omega_1, F_1)$$ and $$(\Omega_2, F_2)$$, respectively. Let \begin{align} X_1:\Omega_1 \rightarrow\mathbb{R}\\ X_2:\Omega_2 \rightarrow\mathbb{R} \end{align} be functions such that $$X_1$$ is measurable with respect to $$(\Omega_1, F_1)$$ and $$X_2$$ is measurable with respect to $$(\Omega_2, F_2)$$.

### Defining a single new space

Define $$\Omega = \Omega_1 \times \Omega_2 = \{(\omega_1, \omega_2) : \omega_1 \in \Omega_1, \omega_2 \in \Omega_2\}$$ Also define $$F$$ as the smallest sigma-algebra on $$\Omega$$ that contains all sets of the form $$A_1 \times A_2$$ such that $$A_1 \in F_1$$, $$A_2 \in F_2$$. (Note 1: Here we define $$\phi \times A_2=A_1\times \phi=\phi$$. Note 2: $$F \neq F_1 \times F_2$$, see example below).

### Fundamental question

Recall that $$\Omega =\Omega_1 \times \Omega_2$$. Does there exist a function $$P:F\rightarrow\mathbb{R}$$ that satisfies $$P[A_1 \times A_2] = P_1[A_1]P_2[A_2] \quad \forall A_1 \in F_1, \forall A_2 \in F_2 \quad (*)$$ and that also satisfies the three axioms of probability with respect to $$(\Omega, F)$$?

This is a deep and hard question, the answer is not obvious. Fortunately, the answer is "yes." Further, the function is unique. This is due to the Hahn-Kolmogorov theorem: https://en.wikipedia.org/wiki/Product_measure

### Consequence of "yes"

Once we have such a function $$P:F\rightarrow\mathbb{R}$$, we have a legitimate new probability space $$(\Omega, F, P)$$. We can define new functions $$X_1^{new}:\Omega\rightarrow\mathbb{R}$$ and $$X_2^{new}:\Omega\rightarrow\mathbb{R}$$ by \begin{align} X_1^{new}(\omega_1, \omega_2) &= X_1(\omega_1) \quad \forall (\omega_1, \omega_2) \in \Omega \\ X_2^{new}(\omega_1, \omega_2) &= X_2(\omega_2)\quad \forall (\omega_1, \omega_2) \in \Omega \end{align} It can be shown that $$X_1^{new}$$ and $$X_2^{new}$$ are both measurable with respect to $$(\Omega, F, P)$$. Thus, they can be called random variables with respect to $$(\Omega, F, P)$$.

We can prove that $$X_1^{new}$$ and $$X_2^{new}$$ are independent: Fix $$x_1, x_2 \in \mathbb{R}$$. Define \begin{align} A_1 &= \{\omega_1 \in \Omega_1 : X_1(\omega_1) \leq x_1\}\\ A_2 &=\{\omega_2 \in \Omega_2 : X_2(\omega_2) \leq x_2\} \end{align} Then \begin{align} &P[X_1^{new} \leq x_1, X_2^{new}\leq x_2] \\ &=P\left[\{\omega \in \Omega: X_1^{new}(\omega) \leq x_1\}\cap \{\omega \in \Omega: X_2^{new}(\omega) \leq x_2\}\right]\\ &= P\left[\{(\omega_1, \omega_2)\in \Omega : X_1(\omega_1)\leq x_1, X_2(\omega_2) \leq x_2\} \right] \\ &= P\left[ A_1 \times A_2 \right]\\ &\overset{(a)}{=} P_1[A_1]P_2[A_2]\\ &\overset{(b)}{=} \left(P_1[A_1]P_2[\Omega_2]\right)\left( P_1[\Omega_1]P_2[A_2]\right)\\ &\overset{(c)}{=} P[A_1 \times \Omega_2]P[\Omega_1 \times A_2]\\ &=P[X_1^{new} \leq x_1]P[X_2^{new}\leq x_2] \end{align} where (a) and (c) hold by the property (*) of the $$P$$ function; (b) holds because $$P_1[\Omega_1]=1$$ and $$P_2[\Omega_2]=1$$. This holds for all $$x_1,x_2 \in \mathbb{R}$$. Thus, $$X_1^{new}$$ and $$X_2^{new}$$ are independent.

### Example to show $$F\neq F_1 \times F_2$$.

Define \begin{align} \Omega_1 &= \{1,2,3\}\\ \Omega_2 &= \{a,b,c\} \\ \Omega &= \Omega_1 \times \Omega_2 \end{align} Define $$F_1$$ and $$F_2$$ as the power sets of $$\Omega_1$$ and $$\Omega_2$$, respectively \begin{align} F_1 &= \{\phi, \{1\}, \{2\}, \{3\}, \{1,2\}, \{1,3\}, \{2,3\}, \{1,2,3\}\}\\ F_2 &= \{\phi, \{a\}, \{b\}, \{c\}, \{a,b\}, \{a,c\}, \{b,c\}, \{a,b,c\}\} \end{align} It can be shown that $$F$$ is the power set of $$\Omega$$. Thus

• $$|F_1 \times F_2| = 8^2 = 64$$.

• $$|\Omega| = 3^2 = 9$$.

• $$|F| = 2^9 = 512$$.

So $$F$$ has more elements than $$F_1 \times F_2$$. The structure of the set $$F_1 \times F_2$$ is also different from that of $$F$$:

• Elements of $$F_1 \times F_2$$ include $$(\phi, \{a\})$$ and $$(\phi, \{b\})$$ and $$(\{1\}, \{a\})$$ and $$(\{2\}, \{b\})$$.

• Elements of $$F$$ include $$\phi$$ and $$\{(1,a), (2,b)\}$$.

### Caveat 1

The set $$F$$ is sometimes called $$F_1 \otimes F_2$$. This is quite different from $$F_1 \times F_2$$, and also different from $$\sigma(F_1 \times F_2)$$.

### Caveat 2

As in my above comments on the question, usually we do not concern ourselves with this deep extension theory.

If we have a probability experiment that involves random variables $$Y$$ and $$Z$$, we implicitly assume there is a single probability space $$(\Omega, F, P)$$ and $$Y:\Omega\rightarrow\mathbb{R}$$ and $$Z:\Omega\rightarrow\mathbb{R}$$ are measurable functions on this space. Thus, for all $$y,z \in \mathbb{R}$$ we know that $$\{Y \leq y\} \in F$$ and $$\{Z \leq z\} \in F$$. Since $$F$$ is a sigma-algebra, this implies that $$\{Y \leq y\}\cap \{Z \leq z\} \in F$$ (for all $$y, z\in \mathbb{R}$$).

The random variables $$Y:\Omega\rightarrow\mathbb{R}$$ and $$Z:\Omega\rightarrow\mathbb{R}$$ are defined to be independent if $$P[Y \leq y, Z\leq z] = P[Y\leq y]P[Z\leq z] \quad \forall y, z \in \mathbb{R}$$

Notice that the definition of independent requires $$\{Y \leq y\} \cap \{Z \leq z\} \in F$$ for all $$y, z \in \mathbb{R}$$, which of course requires $$Y$$ and $$Z$$ to be defined on the same space.

• The independence of $X_1^{new}$ and $X_2^{new}$ only follows because they were constructed such that $X_i^{new}(\omega_1,\omega_2)=X_i^{new}(\omega_i)$ if this were not the case independence is not guaranteed. I think I understand now. Joint Random variables only make sense if each component of the vector RV is defined on the same space be it a product probability space or a prob space? – Noe Vidales May 18 '20 at 20:37
• I don't know what a "joint random variable" is. If you have two random variables $X:\Omega\rightarrow\mathbb{R}$ and $Y:\Omega\rightarrow\mathbb{R}$ you can call $(X,Y)$ a "random vector" if you want. By virtue of $X$ and $Y$ both being random variables on $(\Omega, F, P)$ (with $F$ a sigma-algebra) we have $\{X \leq x\} \cap \{Y\leq y\} \in F$ for all $x,y \in \mathbb{R}$ and so it makes sense to define the "joint CDF" $P[X\leq x, Y\leq y]$. Indeed $X,Y$ must be defined on the same $(\Omega, F, P)$ space in order for probabilities associated with $(X,Y)$ to make sense. – Michael May 18 '20 at 22:30
• Thank you. This has cleared up a lot of my confusions. – Noe Vidales May 18 '20 at 23:20