When can we swap two sigmas in two infinite sums? I have a space of elementary outcomes $\Omega = \{w_1, w_2, w_3, ... \}$
We have assigned a non-negative number $p(w)$ to every elementary outcome $w \in \Omega$ in such a way that
$\sum_{k=1}^\infty p(w_k) = 1$
Can I always swap sigmas in double summations, when both sums are infinite? What about when both sums are finite? Or when we have a finite and an infinite sum there?
In particular, I am mostly interested if in this sum I can swap the two sigmas:
$\sum_{n=1}^\infty  \sum_{k=1}^\infty p(w_k) [w_k \in B_n] \tag{*}$
Here (if it matters) $B_n$ are mutually (pairwise) disjoint events (subsets of $\Omega$), and $p(w_k)$ are these non-negative numbers which are assigned to the elementary outcomes (the elements of $\Omega$).
Also, the Iverson bracket is used there.
Can I swap (exchange) the two sigmas in the sum $(*)$? If so, why? In what cases would I not be allowed to swap the two sigmas?
Note: Sorry for the multiple questions. I just want to understand this matter in details.
 A: Let me try to answer your questions :


*

*Can I always swap sigmas in double summations, when both sums are infinite?
You cannot always do this, for example you can take the sequence defined by,
$$\forall (p,q) \in \Bbb{N}^2, \; a_{p,q} := \delta_{p,q}-\delta_{p+1,q} $$
where $\delta_{i,j} = 1$ when $i =j$ and $0$ otherwise.
If $p \geq 1$ is fixed then $\sum_{q=1}^{\infty}a_{p,q} = a_{p,p}+a_{p,p+1}=1-1=0$
If $q \geq 1$ is fixed :

*

*if $q \geq 2, \sum_{p=1}^{\infty} a_{p,q} = a_{q,q}+a_{q-1,q} = 1-1=0$

*if $q = 1, \sum_{p=1}^{\infty} a_{p,q} = a_{q,q} = 1  $
Therefore,
$$\sum_{p = 1}^{\infty} \sum_{q=1}^{\infty} a_{p,q} = 0 $$
$$\sum_{q = 1}^{\infty} \sum_{p=1}^{\infty} a_{p,q} = 1$$
Hence we cannot swap the sums.
However, when the sequence is non-negative you can always swap the sums. It is also the case when,
$$ \sum_{n=1}^{\infty} \sum_{m=1}^{\infty} |a_{n,m}|<+\infty \quad \text{ or } \quad \sum_{m=1}^{\infty} \sum_{n=1}^{\infty} |a_{n,m}|<+\infty $$
i.e. when $(a_{n,m})$ is summable, cf. down below.


*

*What about when both sums are finite?
You can rearrange them as you want, there is no problem.


*

*Or when we have a finite and an infinite sum there?
Well in that case, if you know that,
$$ \sum_{n = 1}^{\infty} a_n \text{ exists }$$
then it is the exact same thing as manipulating limits :
$$ \sum_{i = 1}^N b_i \sum_{n = 1}^{\infty} a_n = \sum_{n= 1}^{\infty}  \sum_{i = 1}^N b_i a_n $$
However the converse is not true.


*

*Can I swap (exchange) the two sigmas in the sum (∗)? If so, why? In what cases would I not be allowed to swap the two sigmas?
In your case you can perfectly swap the sigmas, because the sequence is non-negative. The best I could do was to give you some insight on what I know. All the following results are from a course I had, named summable families. ($I$ will denote a countable set).

Definition
Let $I$  be a countable set and $(a_i)_{i \in I}$ be a family of non-negative real numbers. The family $(a_i)_{i \in I}$ is said to be summable when,
$$M := \left\{ \sum_{i \in J} a_i : J \text{ finite subset of } I\right\} $$
has an upper bound, in which case we call sum of $(a_i)_{i \in I}$ the real number :
$$\sum_{i \in I} a_i := \sup M$$

The important theorem of this course is the following :

Theorem
Let $(I_{\lambda})_{\lambda \in \Lambda}$  be an at most countable family of subsets of $I$ such that,
$$ I = \bigsqcup_{\lambda \in \Lambda} I_{\lambda}$$
A family $(a_i)_{i \in I}$ of non-negative real numbers is summable if, and only if, for all $\lambda \in \Lambda$, $(a_i)_{i \in I_{\lambda}}$ is summable with sum $S_{\lambda}$ and the family $(S_{\lambda})_{\lambda \in \Lambda}$ is summable. When these conditions are verified, the following equality holds,
$$\sum_{i \in I} a_i=\sum_{\lambda \in \Lambda} S_{\lambda} = \sum_{\lambda \in \Lambda}\left( \sum_{i \in I_{\lambda}} a_i \right) $$

The idea is that you can sum by grouping the terms. However, the proof is rather tedious to write. As a corollary, it yields Fubini's theorem for infinite series in the non-negative case.

Theorem
A doubly-indexed non-negative sequence $(a_{n,m})_{(n,m) \in \Bbb{N}^2}$ is summable if and only if, one of the following equivalent assertions holds :

*

*for all $n \geq 0$, $\sum_m a_{n,m}$ is convergent and $\displaystyle \sum_n \sum_{m =1}^{\infty} a_{n,m}$ is also convergent ;

*for all $m \geq 0$, $\sum_n a_{n,m}$ is convergent and $\displaystyle \sum_m \sum_{n =1}^{\infty} a_{n,m}$ is also convergent.

In which case,
$$\sum_{(n,m) \in \Bbb{N}^2} a_{n,m} = \sum_{n=1}^{\infty} \left(\sum_{m = 1}^{\infty} a_{n,m}\right) = \sum_{m=1}^{\infty} \left(\sum_{n = 1}^{\infty} a_{n,m}\right) $$

Proof :
We apply the previous theorem with $I = \Bbb{N}^2$ and the respective partitions of $I$,

*

*$(I_n^{(1)})_{n \in \Bbb{N}}$ with $I_n^{(1)} = \{(n,m) : m \in \Bbb{N}\} $

*$(I_m^{(2)})_{m \in \Bbb{N}}$ with $I_m^{(2)} = \{(n,m) : n \in \Bbb{N}\} $
If you want a more direct proof you can check this post :
Zeta question - prime zeta. Basic calculus
More generally : a complex-valued family $(a_i)_{i \in I}$ is said to be summable when $(|a_i|)_{i \in I}$ is summable. And in the case of summability, Fubini's theorem holds, namely
$$\sum_{(n,m) \in \Bbb{N}^2} a_{n,m} = \sum_{n=1}^{\infty} \left(\sum_{m = 1}^{\infty} a_{n,m}\right) = \sum_{m=1}^{\infty} \left(\sum_{n = 1}^{\infty} a_{n,m}\right)$$
but in the complex case, there is no such equivalence as in the non-negative one.

I hope this answer can bring some clarity on swapping the sum symbols.
