Rigorous proof of marginalization in probability, i.e. $P_{X}(x) = \sum_{\hat{y} \in \mathcal{Y}} P_{X,Y}(x,\hat{y})$ How do you proof the marginalization rule of probability?
i.e. what is the proof for:
$$P_{X}(x) = \sum_{\hat{y} \in \mathcal{Y}} P_{X,Y}(x,\hat{y})$$
I managed to get a "picture proof" by drawing a venn diagram and then looking at the following equation:
$$ P_{X}(x) = \sum_{\hat{y} \in \mathcal{Y}} P_{X,Y}(x,\hat{y}) = P_{X,Y}(x,y) + P_{X,Y}(x,\bar{y}) $$
Which by inspecting the venn diagram:

One can notice that translating $P_{X,Y}(x,y) + P_{X,Y}(x,\bar{y})$ into sets gives:
$$X \cap Y$$
and
$$X \cap \bar{Y}$$
Which covers the whole "area" of X. I guess this makes sense in this case, but I feel that there are some issues with the "proof".

*

*Its a proof by picture (not a real proof)


*Does not generalize very well for an alphabet size of more values or more random variables.


*Not sure how to generalize this for continuous random variables


*Does not feel rigorous enough (probably because of the previous reasons).
I was wondering, is this just an axiom of probability or can it be derived from more basic axioms? I am having a hard time generalizing this.
Also, I thought this would have been a basic result in probability and should be in the web bus was unable to find any good rigorous reference. If possible I'd like an official reference too e.g. a textbook.
 A: Apply one of the axioms of a probability measure to the disjoint union of events $$[X=x]=\bigcup_{y}[X=x,Y=y].$$
A: Denote $B_j \in \Omega$ as disjoint events such that $\bigcup_{j=1}^n B_j = \Omega$.
For any measurable $\{X \in A\}$ we will clearly have:
$$\{X\in A\} = \bigcup_{j=1}^n\{X\in A, Y\in B_j\}$$
Each event $\{X\in A, Y \in B_j\}$ is disjoint since $B_j$ is.
Hence from Kolmogorovs third axiom for countable sequences of disjoint sets
$$\mathbb{P}\left(\bigcup_{j=1}^n\{X\in A, Y\in B_j\}\right) = \sum_{j=1}^n \mathbb{P}(X \in A, Y \in B_j) $$
Therefore
$$\mathbb{P}(X \in A) = \sum_{j=1}^n \mathbb{P}(X \in A, Y \in B_j)$$
for any disjoint $B_j$ such that $\bigcup B_j = \Omega$
Kolmogorovs third axiom is $P(\bigcup_{j=1}^{\infty}B_j) = \sum_{j=1}^{\infty} P(B_j)$ for disjoint $B_j$. For a finite number of events we just use finite additivity which can be derived by setting each $B_{n+1}, B_{n+1},\dotsc = \emptyset$ since all empty sets are disjoint:    $\emptyset \cup \emptyset = \emptyset$.
A: A set of joint random variables generally falls in 3 categories:
1- Continuous (the CDF $F_{X_1\cdots X_n}(x_1,\cdots,x_n)$ is continuous over $\Bbb R^n$),
2- Discrete (the CDF $F_{X_1\cdots X_n}(x_1,\cdots,x_n)$ is piecewise-constant and the PDF $f_{X_1\cdots X_n}(x_1,\cdots,x_n)$ only contains Kronecker's delta),
3- Mixed (discrete over some parts of the domain of their definition and continuous elsewhere).
There are many examples for any of the categories. Let us consider some for two joint variables:
1- Continuous example: two independent exponential random variables with $\lambda=1$
$$
F_{XY}(x,y)=\Pr\{X\le x,Y\le y\}=\begin{cases}(1-e^{-x})(1-e^{-y})&,\quad (x,y)\in\Bbb R^+\times \Bbb R^+\\
0&,\quad \text{elsewhere}\end{cases}
$$
2- Discrete example: two independent Bernoulli random variables with $p=\frac{1}{2}$
$$
F_{XY}(x,y)=\Pr\{X\le x,Y\le y\}=\begin{cases}
0.25&,\quad 0\le x<1\ \ , \ \ 0\le y<1\\
0.5&,\quad 0\le x<1\ \ , \ \ 1\le y\\
0.5&,\quad 0\le y<1\ \ , \ \ 1\le x\\
1&,\quad 1\le x\ \ , \ \ 1\le y\\
0&,\quad\text{elsewhere}
\end{cases}
$$
for which
$$
f_{XY}(x,y)=0.25\delta_{0,0}(x,y)+0.25\delta_{1,0}(x,y)+0.25\delta_{0,1}(x,y)+0.25\delta_{1,1}(x,y),
$$
where $\delta_{\alpha,\beta}(x,y)$ is the Kronecker's delta at $(x,y)=(\alpha,\beta)$.
3- Mixed example: two independent random variables $X$ and $Y$, such that
$$
F_X(x)=\Pr\{X\le x\}=\begin{cases}
0&,\quad x<0\\
1-\frac{1}{2}\exp(-x)&,\quad x\ge 0
\end{cases},
$$
so that
$$
f_X(x)=0.5\delta_0(x)+0.5\exp(-x)I_{[0,\infty)}(x).
$$
We hence prove the matter of question for a general CDF $\Pr\{X_1\le x_1,\cdots,X_n\le x_n\}$. Note that according to the definition of PDF, we have
$$
f_{X_1\cdots X_n}(x_1,\cdots,x_n)=\frac{d^n F_{X_1\cdots X_n}(x_1,\cdots,x_n)}{dx_1\cdots dx_n},
$$
which, according to the fact that $F_{X_1\cdots X_n}(x_1,\cdots,x_n)=0$ is at least one of the $\{x_i\}_{i=1}^n$ is equal to $-\infty$, implies
$$
F_{X_1\cdots X_n}(x_1,\cdots,x_n)=
\int_{-\infty}^{x_1}
\cdots
\int_{-\infty}^{x_n}
f_{X_1\cdots X_n}(u_1,\cdots,u_n)
du_n\cdots du_1.
$$
Also,
$$
F_{X_1\cdots X_n}(x_1,\cdots,x_n)=\Pr\{X_1\le x_1,\cdots,X_n\le x_n\},
$$
and
$$
\Pr\{X_1\le x_1\}{=\Pr\{X_1\le x_1,X_2\le \infty,\cdots,X_n\le \infty\}
\\=F_{X_1\cdots X_n}(x_1,\infty,\cdots,\infty)
}
.
$$
From these implications, we conclude that
$$
\Pr\{X_1\le x_1\}=F_{X_1\cdots X_n}(x_1,\infty,\cdots,\infty)
\\\implies
\Pr\{X_1\le x_1\}=
\int_{-\infty}^{x_1}
\int_{-\infty}^{\infty}
\cdots
\int_{-\infty}^{\infty}
f_{X_1\cdots X_n}(u_1,\cdots,u_n)
du_n\cdots du_2du_1
\\\implies
\frac{d}{dx_1}\Pr\{X_1\le x_1\}=
\frac{d}{dx_1}
\int_{-\infty}^{x_1}
\int_{-\infty}^{\infty}
\cdots
\int_{-\infty}^{\infty}
f_{X_1\cdots X_n}(u_1,\cdots,u_n)
du_n\cdots du_2du_1
\\\implies
f_{X_1}(x_1)=
\int_{-\infty}^{\infty}
\cdots
\int_{-\infty}^{\infty}
\frac{d}{dx_1}
\int_{-\infty}^{x_1}
f_{X_1\cdots X_n}(u_1,\cdots,u_n)
du_1du_n\cdots du_2
\\\implies
f_{X_1}(x_1)=
\int_{-\infty}^{\infty}
\cdots
\int_{-\infty}^{\infty}
f_{X_1\cdots X_n}(x_1,\cdots,u_n)
du_n\cdots du_2,
$$thus completing the proof. For the last equality, we used
$$
\frac{d}{dx}\int_a^x f(u)du=f(x)-f(a)
$$
and $f_{X_1\cdots X_n}(-\infty,\cdots,u_n)=0$ $\blacksquare$
A: I shall show marginalization over Lebesgue densities.
Consider $P((X,Y)\in B), B\in \mathscr{B}(\mathbb{R}^2)$ and define $\mu_{(X,Y)}(B):=P((X,Y)\in B)$. Then this is a probability measure on $(\mathbb{R}^2,\mathscr{B}(\mathbb{R}^2))$, the joint distribution of $X,Y$. To see this, note
$$\{(X,Y)\in C\times D\}=\{X\in C\}\cap  \{Y\in D\}\in \mathscr{F}
$$
Since the rectangle sets generate $\mathscr{B}(\mathbb{R}^2)$, $(X,Y):\Omega \to \mathbb{R}^2$ is a measurable function so its image measure is a probability measure.
We shall prove $\mu_{(X,Y)}(A\times \mathbb{R})=\mu_X(A),\forall A\in\mathscr{B}(\mathbb{R})$ where $\mu_X$ is the distribution of $X$. Indeed:
$$\mu_{(X,Y)}(A\times \mathbb{R})=P(\{X\in A\}\cap \Omega\}=P(X\in A)=:\mu_X(A)$$
Now if $\mu_{(X,Y)}$ has a Lebesgue density, by Fubini-Tonelli we have
$$\mu_X(A)=\int_{A\times \mathbb{R}}f_{(X,Y)}dz=\int_A\int_{\mathbb{R}}f_{(X,Y)}(x,y)dydx$$
And if $\mu_X$ also has a density this implies $\int_Af_Xdx=\int_A(\int_{\mathbb{R}}f_{(X,Y)}(x,y)dy)dx$ for all Borel $A$ so
$$f_X(x)=\int_\mathbb{R}f_{(X,Y)}(x,y)dy$$
for $dx$-almost all $x$.
A: The ideas used for the multivariate case follow the same idea as the bivariate case, but they can be harder to follow. Therefore, I'll treat this particular case below in order to make the ideas behind the proof clearer.
Claim
Let $(X,Y)$ be a bivariate discrete random variable and let $A$ be a subset of $\Omega$ such that $\mathbb{P}(X=x , Y=y) > 0$ for all $y \in A$. Then, $\mathbb{P}(X=x) = \sum _{y \in A} \mathbb{P}(X=x , Y=y)$.
Proof
First notice that the probability of {$X = x$} for a particular choice of $x$, is exactly the same as the probability of {$X=x$ and $Y \in A$}  since $Y$ will always take values in $A$ and so this doesn't affect the probability of the event. This could similarly be written as the probability of the event {there exists $y \in A$ such that $X=x$ and $Y=y$} as this event will be true if and only if {$X=x$} (as, again, $Y$ is always in $A$).
Let's write this up formally so that we can now manipulate this in the direction of the desired result.
$$\mathbb{P}(X=x) \space = \space \mathbb{P}(\{\exists y \in A : X=x , Y=y \})$$
Since we are looking at the probability that there exists at least one $y \in A$, this is equivalent to taking a union over all possible values of $y \in A$. Therefore, we can reduce the above result to:
$$ \mathbb{P}(\{\exists y \in A : X=x , Y=y \}) \space = \space \mathbb{P}(\cup _{y \in A}(X=x , Y=y)) $$
Now that we have a union within a probability, we can simply apply the countable additivity (which is a property of all probability measures) to arrive at the result:
$$\mathbb{P}(\cup _{y \in A}(X=x , Y=y)) \space = \space \sum _{y \in A} \mathbb{P} (X=x, Y=y)$$
Since $Y$ is equal to $0$ outside of the set $A$, we can replace the summation index with the sample space $\Omega$ to conclude the proof.
