Interchange summation (outer infinite, inner dependent on outer)

For finite summation limits, I believe that the following holds (for some general function $f$): $\sum_{i=2}^n \sum_{j=1}^{i-1} f(i,j) = \sum_{j=1}^{n-1} \sum_{i=j+1}^{n} f(i,j)$ ... (1)

However, I'm having trouble reasoning about this in the infinite case. Does the following hold? (Again, let's assume $f$ is some general function, here also assuming that the summation converges.) $\sum_{i=2}^{\infty} \sum_{j=1}^{i-1} f(i,j) = \sum_{j=1}^{\infty} \sum_{i=j+1}^{\infty} f(i,j)$ ... (2)

My trouble is that I don't know how to reason about the infinite bounds. Basically, we're summing $f(i,j)$ across all $i,j$ pairs with $i>j$. The LHS inner summation upper bound $i-1$ and the RHS inner summation lower bound $j+1$ ensure $j<i$ for all $f(i,j)$ terms summed. But how can I reason about things like $i-1$ and $j+1$ when $i$ or $j$ can reach $\infty$?

Ultimately, in practice, am I allowed to write expressions like Equation 2? I'm an engineer, so is the finer point about $\infty$ something I should let mathematicians think about?

• If the overall sum converges absolutely, then this works. (This can be regarded either as rearrangement of absolutely convergent sums, or as a special case of Fubini's theorem.) But there are some famous counterexamples where the equality fails. – Ian Feb 11 '16 at 23:03

The infinite sums in (2) are different ways of writing the sum of $f(i,j)$ over the set $\{(i,j):i>j,j\ge 1\}$. For non-negative numbers $f(i,j)$ all ways of rearranging such a sum converge if any of them converges, and they all converge to the same thing. The same is true if the sum of their absolute values converges. This is Baby Rudin 3.55.
On the other hand, if the sum of their absolute values diverges, then all bets are off as shown by the examples in Baby Rudin 3.53. For example it is famously known that $\sum_{n=0}^\infty (-1)^n/n$ converges, with negative terms cancelling the positive ones; but $\sum_{n=0}^\infty 1/n$ doesn't, and you can rearrange the terms in $\sum_{n=0}^\infty (-1)^n/n$ to reach a variety of different sums depending on how you rearrange them.
Thus, equation (2) may or may not be correct for general $f(i,j)$. An easy way to test whether it works is to see if one side converges replacing $f(i,j)$ by $|f(i,j)|$, so that you determine that the sum converges absolutely. If this test fails, (2) may still be correct but then you have to reason about it in a rather groady way.
You are asking "how can I reason about things like $i−1$ and $j+1$ when $i$ or $j$ can reach $∞$?" The short answer is, they don't actually "reach $\infty$"; instead they traverse the whole set of natural numbers (with $i>j$), and for natural numbers $i-1$ and $j+1$ are always well defined.