Why do we believe that $\sum_{k=1}^{\infty} x_k=\sum_{i=1}^{\infty}\sum_{j=1}^{\infty}x_{ij}$? I have an absolutely convergent series $\sum_{k=1}^{\infty}x_k=s$. I manage somehow to index all elements of the series along two dimensions, so each element of the series $x_{k}$ is associated with a pair $(i,j)$ and $i,j \in \mathbb N$ through a bijective mapping, so I can refer to it as $x_{i,j}$ now. 
It is clear that any rearrangement of the series absolutely converges to $s$. But why can we claim that (at least my book does so) $\sum_{k=1}^{\infty} x_k=\sum_{i=1}^{\infty}\sum_{j=1}^{\infty}x_{i,j}$. 
Expression $\sum_{i=1}^{\infty}\sum_{j=1}^{\infty}x_{i,j}$ is not just a rearrangement (is not it?), it is a series (over $i$) of an infinite number of series (each over $j$). In a sense it is a series of limits of many sequences of partial sums. Why should we believe that it converges to the same $s$. Am I lacking knowledge of a theorem here? 
 A: As pointed out by Reveillark, this is in fact a particular case of the Fubini's theorem. In this case, however, we need not get bogged down by its general setting which involves product measure things (in our case, product of two counting measures). We have a simpler proof.
Let us introduce four notations
$$ s(m, n) = \sum_{i=1}^{m}\sum_{j=1}^{n} x_{i,j},
\quad s(\infty, n) = \sum_{i=1}^{\infty}\sum_{j=1}^{n} x_{i,j},
\quad s(m, \infty) = \sum_{i=1}^{m}\sum_{j=1}^{\infty} x_{i,j} $$
and finally
$$ s(\infty, \infty) = \sum_{k=1}^{\infty} x_k. $$
Here, symbols $s(\infty, n)$ and $s(m, \infty)$ are well-defined since
$$ \forall j, \ \sum_{i=1}^{\infty} |x_{i,j}| \leq \sum_{k=1}^{\infty} |x_k| < \infty, \quad \text{and} \quad \forall i, \ \sum_{j=1}^{\infty} |x_{i,j}| \leq \sum_{k=1}^{\infty} |x_k| < \infty. $$
Then we find that


*

*From the rearrangement theorem, we know that $s(m_j, n_j) \to s(\infty, \infty)$ for any pair of indexing sequences $(m_j)$ and $(n_j)$ tending to infinity.

*Fix $\epsilon > 0$. Then there exists $m_1 < m_2 < \cdots$ such that for each $n$,
$$|s(m, n) - s(\infty, n)| < \epsilon \quad \text{whenever} \ m \geq m_n. $$
Consequently,
\begin{align*}
|s(\infty, \infty) - s(\infty, n)|
&\leq |s(\infty, \infty) - s(m_n, n)| + |s(m_n, n) - s(\infty, n)| \\
&\leq |s(\infty, \infty) - s(m_n, n)| + \epsilon.
\end{align*}
and taking $\limsup$ as $n\to\infty$,
$$ \limsup_{n\to\infty} |s(\infty, \infty) - s(\infty, n)| \leq \epsilon. $$
But since $\epsilon > 0$ was arbitrary, this implies that the limsup equals 0. Therefore $s(\infty, n)$ converges to $s(\infty, \infty)$ as desired.

*Similar argument shows that $s(m, \infty)$ converges to $s(\infty, \infty)$.
A: No, its not a rearrangement in the conventional sense. We need some additional definitions and theorems to interpret and prove this.
A double sequence is a function whose domain is the set $\mathbb{N} \times \mathbb{N}$. We denote the image of the pair $(i,j)$ by $x_{i,j}$. 
We say that $\displaystyle \lim_{i,j \to \infty}x_{i,j}=L$ if and only if for every $\varepsilon>0$ there exists $N$ such that $i>N$ and $j>N$ implies $|x_{i,j}-L|<\varepsilon$. This is called a double limit.
A double sequence induces a sequence $a_j=\displaystyle\lim_{i \to \infty} x_{i,j}$. As such, this sequence may have some limit. It is natural to wonder whether: 
$$\lim_{i,j \to \infty}x_{i,j}=\lim_{j \to \infty} \left ( \lim_{i \to \infty} x_{ij} \right )=\lim_{i \to \infty} \left ( \lim_{j \to \infty} x_{i,j} \right )$$
If we know that the sequence has double limit $L$ and that, for each fixed $i$, $\displaystyle \lim_{j \to \infty}x_{i,j}$ exists, then $\displaystyle \lim_{i \to \infty} \left ( \lim_{j \to \infty} x_{i,j} \right )$
Given a double sequence $x_{mn}$, we can define a new sequence by: $$s(p,q)=\sum_{m=1}^p\sum_{n=1}^qx_{m,n}$$
This is called a double series. 
Another definition: Given a double sequences $x_{i,j}$ and a bijective mapping $g: \mathbb{N} \rightarrow \mathbb{N} \times \mathbb{N}$, define a sequence by:
$$G(n)=x_{g(n)}$$
The function $g$ is called a rearragnment of the double sequence $x_{i,j}$ into the sequence {$G$}. 
We finally have the following result:
Theorem: Let $\sum x_{m,n}$ be a given double series and $g$ a rearrangement into the sequence $G$. Then:
a) $\sum_{n=1}^\infty G(n)$ converges absolutely if and only if $\sum_{n,m=1}^\infty x_{nm}$ converges absolutely. 
If, addiotionally, $\sum_{n,m=1}^\infty x_{nm}$ converges absolutely with sum $S$ we have that:
b) $\sum_{n=1}^\infty G(n)=S$
c) $\sum_{n=1}^\infty x_{n,m}$ and $\sum_{m=1}^\infty x_{n,m}$ are both absolutely convergent.
d) If $A_m=\sum_{n=1}^\infty x_{n,m}$ and $B_n=\sum_{m=1}^\infty x_{n,m}$ then the series $\sum_{n=1}^\infty A_n$ and $\sum_{m=1}^\infty B_m$ are both absolutely convergent with sum $S$. This is: $$\sum_{m=1}^\infty \left ( \sum_{n=1}^\infty x_{n,m} \right )=\sum_{n=1}^\infty \left ( \sum_{m=1}^\infty x_{n,m} \right )
=\sum_{n,m=1}^\infty x_{nm}=S$$
For the proof of this result (its rather long so typing it up would be begging for me to mess up the indices), and some other concerning sufficient conditions for the convergence of the double series, see Apostol's Mathematical Analysis
A: At the end of the day, when you have absolute convergence you can do whatever crazy rearrangement you want, including spreading the terms of one infinite sum across multiple infinite sums and so forth. Here's a perspective on why that fact boils down to the more basic fact about rearrangements that you seem to be familiar with.
Start out with $\{ x_j \}_{j=1}^\infty$, a sequence which is absolutely summable. Identify this sequence with part of a double sequence by defining $y_{1,j}=x_j$. Then you extend $y$ into a double sequence by defining $y_{i,j}=0$ for $i=2,3,\dots$ and all $j$. Then I claim that
$$\sum_{j=1}^\infty x_j = \sum_{i=1}^\infty \sum_{j=1}^\infty y_{i,j}.$$
This is because the sequence $z_n = \sum_{i=1}^n \sum_{j=1}^\infty y_{i,j}$ is identically equal to $\sum_{j=1}^n x_j$, so $\lim_{n \to \infty} z_n = \sum_{j=1}^\infty x_j$. On the other hand $\lim_{n \to \infty} z_n = \sum_{i=1}^n \sum_{j=1}^\infty y_{i,j}$ by the definition of the right side.
Now that you've done that, you can get your result by rearranging $\sum_{i=1}^\infty \sum_{j=1}^\infty y_{i,j}$.
