Let me try to answer your questions :
- Can I always swap sigmas in double summations, when both sums are infinite?
You cannot always do this, for example you can take the sequence defined by,
$$\forall (p,q) \in \Bbb{N}^2, \; a_{p,q} := \delta_{p,q}-\delta_{p+1,q} $$
where $\delta_{i,j} = 1$ when $i =j$ and $0$ otherwise.
If $p \geq 1$ is fixed then $\sum_{q=1}^{\infty}a_{p,q} = a_{p,p}+a_{p,p+1}=1-1=0$
If $q \geq 1$ is fixed :
- if $q \geq 2, \sum_{p=1}^{\infty} a_{p,q} = a_{q,q}+a_{q-1,q} = 1-1=0$
- if $q = 1, \sum_{p=1}^{\infty} a_{p,q} = a_{q,q} = 1 $
Therefore,
$$\sum_{p = 1}^{\infty} \sum_{q=1}^{\infty} a_{p,q} = 0 $$
$$\sum_{q = 1}^{\infty} \sum_{p=1}^{\infty} a_{p,q} = 1$$
Hence we cannot swap the sums.
However, when the sequence is non-negative you can always swap the sums. It is also the case when,
$$ \sum_{n=1}^{\infty} \sum_{m=1}^{\infty} |a_{n,m}|<+\infty \quad \text{ or } \quad \sum_{m=1}^{\infty} \sum_{n=1}^{\infty} |a_{n,m}|<+\infty $$
i.e. when $(a_{n,m})$ is summable, cf. down below.
- What about when both sums are finite?
You can rearrange them as you want, there is no problem.
- Or when we have a finite and an infinite sum there?
Well in that case, if you know that,
$$ \sum_{n = 1}^{\infty} a_n \text{ exists }$$
then it is the exact same thing as manipulating limits :
$$ \sum_{i = 1}^N b_i \sum_{n = 1}^{\infty} a_n = \sum_{n= 1}^{\infty} \sum_{i = 1}^N b_i a_n $$
However the converse is not true.
- Can I swap (exchange) the two sigmas in the sum (∗)? If so, why? In what cases would I not be allowed to swap the two sigmas?
In your case you can perfectly swap the sigmas, because the sequence is non-negative. The best I could do was to give you some insight on what I know. All the following results are from a course I had, named summable families. ($I$ will denote a countable set).
Definition
Let $I$ be a countable set and $(a_i)_{i \in I}$ be a family of non-negative real numbers. The family $(a_i)_{i \in I}$ is said to be summable when,
$$M := \left\{ \sum_{i \in J} a_i : J \text{ finite subset of } I\right\} $$
has an upper bound, in which case we call sum of $(a_i)_{i \in I}$ the real number :
$$\sum_{i \in I} a_i := \sup M$$
The important theorem of this course is the following :
Theorem
Let $(I_{\lambda})_{\lambda \in \Lambda}$ be an at most countable family of subsets of $I$ such that,
$$ I = \bigsqcup_{\lambda \in \Lambda} I_{\lambda}$$
A family $(a_i)_{i \in I}$ of non-negative real numbers is summable if, and only if, for all $\lambda \in \Lambda$, $(a_i)_{i \in I_{\lambda}}$ is summable with sum $S_{\lambda}$ and the family $(S_{\lambda})_{\lambda \in \Lambda}$ is summable. When these conditions are verified, the following equality holds,
$$\sum_{i \in I} a_i=\sum_{\lambda \in \Lambda} S_{\lambda} = \sum_{\lambda \in \Lambda}\left( \sum_{i \in I_{\lambda}} a_i \right) $$
The idea is that you can sum by grouping the terms. However, the proof is rather tedious to write. As a corollary, it yields Fubini's theorem for infinite series in the non-negative case.
Theorem
A doubly-indexed non-negative sequence $(a_{n,m})_{(n,m) \in \Bbb{N}^2}$ is summable if and only if, one of the following equivalent assertions holds :
- for all $n \geq 0$, $\sum_m a_{n,m}$ is convergent and $\displaystyle \sum_n \sum_{m =1}^{\infty} a_{n,m}$ is also convergent ;
- for all $m \geq 0$, $\sum_n a_{n,m}$ is convergent and $\displaystyle \sum_m \sum_{n =1}^{\infty} a_{n,m}$ is also convergent.
In which case,
$$\sum_{(n,m) \in \Bbb{N}^2} a_{n,m} = \sum_{n=1}^{\infty} \left(\sum_{m = 1}^{\infty} a_{n,m}\right) = \sum_{m=1}^{\infty} \left(\sum_{n = 1}^{\infty} a_{n,m}\right) $$
Proof :
We apply the previous theorem with $I = \Bbb{N}^2$ and the respective partitions of $I$,
- $(I_n^{(1)})_{n \in \Bbb{N}}$ with $I_n^{(1)} = \{(n,m) : m \in \Bbb{N}\} $
- $(I_m^{(2)})_{m \in \Bbb{N}}$ with $I_m^{(2)} = \{(n,m) : n \in \Bbb{N}\} $
If you want a more direct proof you can check this post :
Zeta question - prime zeta. Basic calculus
More generally : a complex-valued family $(a_i)_{i \in I}$ is said to be summable when $(|a_i|)_{i \in I}$ is summable. And in the case of summability, Fubini's theorem holds, namely
$$\sum_{(n,m) \in \Bbb{N}^2} a_{n,m} = \sum_{n=1}^{\infty} \left(\sum_{m = 1}^{\infty} a_{n,m}\right) = \sum_{m=1}^{\infty} \left(\sum_{n = 1}^{\infty} a_{n,m}\right)$$
but in the complex case, there is no such equivalence as in the non-negative one.
I hope this answer can bring some clarity on swapping the sum symbols.
