$\sum_{i=1}^{\infty}\sum_{j=1}^{\infty} |a_{ij}|$ converges if and only if $\sum_{j=1}^{\infty}\sum_{i=1}^{\infty} |a_{ij}|$ converges? Let $\{a_{ij}: i, j \in \mathbf{N}\}$ be a doubly indexed array of real numbers. Is it true that $\sum_{i=1}^{\infty}\sum_{j=1}^{\infty} |a_{ij}|$ converges if and only if $\sum_{j=1}^{\infty}\sum_{i=1}^{\infty} |a_{ij}|$ converges? If so, how can I prove it?
 A: If you are satisfied with simply invoking Tonelli's theorem then there is nothing to do. If you want a proof , then ...
If $\sum_{i=1}^\infty \sum_{j=1}^\infty|a_{ij}| = S < \infty$, then for all $i$ the series $A_i = \sum_{j=1}^\infty |a_{ij}|$ converges and $\sum_{i=1}^\infty A_i = S$.
Since $|a_{ij}| \leqslant A_i$ for all $j$ it follows that $\sum_{i=1}^\infty |a_{ij}|$ converges by the comparison test.
Thus, for all $N$, since the limit of a finite sum is the sum of the limits,
$$S_N = \sum_{j=1}^N \sum_{i=1}^\infty|a_{ij}| = \sum_{i=1}^\infty \sum_{j=1}^N|a_{ij}| \leqslant \sum_{i=1}^\infty \sum_{j=1}^\infty|a_{ij}| = S $$
Since $S_N$ is non-decreasing and bounded, it converges and
$$\lim_{N \to \infty} S_N = \sum_{j=1}^\infty \sum_{i=1}^\infty|a_{ij}| \leqslant S.$$
Reversing the argument we can also show that the inequliaty on the RHS can be replaced with equality.
A: In fact, the following series version of the Tonelli's theorem holds:

Proposition. For any double sequence $(a_{i,j})$ of non-negative real numbers, we have
$$ \sum_{i=1}^{\infty}\sum_{j=1}^{\infty} a_{i,j} = \sum_{j=1}^{\infty}\sum_{i=1}^{\infty} a_{i,j}. $$

This holds when one of the sums is $+\infty$, in which case both sides equal $+\infty$ as well. The proof is also straightforward:
$$\sum_{i=1}^{\infty}\sum_{j=1}^{\infty} a_{i,j}
= \sup_{m\geq1}\sum_{i=1}^{m} \left( \sup_{n\geq1}  \sum_{j=1}^{n} a_{i,j} \right)
= \sup_{m\geq1} \sup_{n\geq1} \sum_{i=1}^{m} \sum_{j=1}^{n} a_{i,j}
= \sup_{(m,n)\in\mathbb{N}^2} \sum_{i=1}^{m} \sum_{j=1}^{n} a_{i,j} $$
Here are comments on each equality:


*

*In the second equality, the identity $\sup(A+B) = \sup A+ \sup B$ is utilized. Finally

*In the first equality, the following general fact is utilized:

Lemma. Let $A$ and $B$ be non-empty sets and $f:A\times B \to \mathbb{R}$ be a function. Then
$$\sup_{x\in A}\sup_{y\in B}f(x, y) = \sup_{(x,y)\in A\times B}f(x, y) = \sup_{y\in B}\sup_{x \in A} f(x, y). $$

Exactly the same argument shows that
$$\sum_{j=1}^{\infty}\sum_{i=1}^{\infty} a_{i,j}
= \sup_{(m,n)\in\mathbb{N}^2} \sum_{j=1}^{n} \sum_{i=1}^{m} a_{i,j}, $$
hence the conclusion follows from the fact that the order of iterated summations can be switched for finite sums.

Addendum. As a side note, this proof demonstrated how powerful monotonicity is. As an exercise, you can prove the following special case of Abelian theorem only by modifying the proof above a bit:

Proposition. For any sequence $(a_n)_{n\geq 0}$ of non-negative real numbers, the following holds:
$$\sum_{n=0}^{\infty} a_n = \lim_{x\to 1^-} \sum_{n=0}^{\infty} a_n x^n. $$

