What is the square root of infinity and what is infinity^2? In rigorous mathematics, how can one formalize the notion of a "square root of infinity", as well as that of "infinity squared"?
 A: It is not quite correct to write that "$\infty$ is not a number", as in the accepted answer.  In fact, John Wallis who invented the symbol used it precisely as an infinite number, and its reciprocal $\frac{1}{\infty}$ as an infinitesimal number. Rather, the convention among mathematicians today happens to be that the symbol $\infty$ is interpreted in different ways, but such an outcome was by no means pre-determined in any sense.
While one can use ordinal numbers, with some effort, to construct a surreal
extension of the real numbers that admits basic operations like squares and
roots, it is conceptually easier to use a hyperreal number field, ${}^\ast
\mathbb R$.
Such a field ${}^\ast \mathbb R$ is similarly an extension of the reals, but
is a more powerful tool in analysis.  This is because not merely the squaring
function and the root function can be extended, but in fact all functions
can be extended to ${}^\ast \mathbb R$, and the extension is fairly automatic
once one has a hyperreal field in place.
By contrast, even such a simple function as the sine function has so far
resisted all efforts of extension to the surreals.
One way to construct ${}^\ast \mathbb R$ is as a quotient of the ring of
sequences, $\mathbb R^{\mathbb N}$, by a suitable maximal ideal MAX, so that
${}^\ast \mathbb R=\mathbb R^{\mathbb N}/\,\text{MAX}$.
Thus both the "square root of infinity" and "square of infinity" make sense when infinity is interpreted as a hyperreal number. An example of an infinite number in ${}^\ast \mathbb R$ is represented by the sequence $1,2,3,\ldots$.
One way of constructing MAX is to use a finitely additive measure $\xi$ on subsets of $\mathbb{N}$ with values $0$ or $1$. Then a sequence is in MAX if it vanishes for a set of indices of measure $1$.
A: That really depends on what you mean by "infinity". If you mean $\infty$, then that is not a number, but rather a shorthand for the concept that some quantity (usually a natural or real number) grows beyond any finite bound. As such, you cannot multiply it with anything, and especially not itself. There are, however, several arithmetic systems that has elements larger than any finite sum of the form $1+1+\cdots +1$, and thus deserve to be called infinite in size. I'll tell you about three of them (slightly simplified, but hopefully not directly incorrect).

The first one is the cardinals. They signify how large something (a set) is. A finite cardinal is just a natural number (signifying "the size of a set with that many elements"), but there are also ininite cardinals. The smallest infinite cardinal is $\aleph_0$, the size of the set of natrual numbers.
Addition of cardinals works the way you would expect addition of sizes to work, namely putting the two sets next to one another, and count how many elements there are in total. More specifically, if you have two cardinal numbers $\kappa_1, \kappa_2$, each signifying the size of two sets $X_1, X_2$, then the cardinal $\kappa_1+\kappa_2$ is the cardinality of the disjoint union $X_1\sqcup X_2$, or equivalently, the set of pairs $(x_i, i)$ where $x_i \in X_i$ and $i \in \{1, 2\}$.
Multiplication of cardinals work the following way: $\kappa_1\cdot\kappa_2$ is the size of the set $X_1\times X_2$, the set of pairs $x_1, x_2$ with $x_1\in X_1$ and $x_2\in X_2$. If the largest of $\kappa_1$ and $\kappa_2$ is infinite, then $\kappa_1+\kappa_2 = \kappa_1\cdot\kappa_2 = \max(\kappa_1, \kappa_2)$. This means that if $\kappa$ is an infinite cardinal, $\kappa^2 = \kappa$, so we also get $\sqrt\kappa = \kappa$.
(You can also define exponents: $\kappa_1^{\kappa_2}$ is the size of the set of all possible functions form $X_2$ to $X_1$. For instance, $2$ is a two-element set, so $\kappa^2$ is the set of functions from a two element set to $\kappa$. A function from a two-element set is the same as an ordered pair, so this is actually the same as $\kappa\cdot \kappa$. Neat, huh?)

The second one is the ordinals. They signify orderings of objects. However, not all orderings, but orderings where any subset has a smallest element, so-called well-orderings. Again, a finite ordinal is just a natural number (signifying "the ordering of all smaller natural numbers"), but just as last time, there are infinite ordinals, the smallest of which is called $\omega_0$, or just $\omega$, and it signifies the ordering of the natural numbers.
Addition of ordinals is done the following way: If $\gamma, \lambda$ are ordinals, then $\gamma + \lambda$ is the ordering gotten by putting $\gamma$ in front of $\lambda$. For instance, $1 + \omega$ is the same as $\omega$, because if you take the natural numbers, and put one element in front of all of them, you have something that as far as ordering is concerned, looks exactly the same as the natural numbers themselves. However, $\omega + 1$ means putting a single element after all the natural numbers, which is a different order.
Multiplication works the following way: $\gamma\cdot \lambda$ is the ordinal we get by taking $\lambda$, substituting every element in that ordering by a copy of $\gamma$, and then add them all in that order (i.e. put them after one another) (we specify that you work your way from left to right). That way, $2\cdot \omega$ means taking the natural numbers, swapping each of the numbers there for two numbers, and then put all these pairs after one another. This gives us $\omega$ back. However, $\omega\cdot 2$ means taking an ordered pair, swap each of the two elements with a copy of the natural numbers, and then putting one copy after the other. This is the same as you would get by calculating $\omega + \omega$.
In this framework, multiplication and addition of infinite ordinals is not as trivial as for the cardinals. We get, for instance, $\omega\cdot \omega = \omega + \omega +\omega +  \cdots $, which is the smallest infinite perfect square ordinal. As with the natural numbers themselves, there are some ordinals that have a square root, and some that do not. Specifically, $\omega$ does not have a square root.
(You can define exponents for ordinals as well: In this case, $\gamma^\lambda$ is the ordinal we get if we take $\lambda$, replace every element in it with copies of $\gamma$, and multiply them all together, just like multiplication was defined as repeated addition. This makes $\omega^2 = \omega\cdot \omega$. Neat, huh? Note that while ordinal and cardinal addition and multiplication are somewhat similar, their notions of exponentiation are very different.)

Lastly, I'll tell you about the surreal numbers. While ordinals and cardinals are in heavy use in set theory, the surreal numbers are more of a curiosity. They are also a bit more difficult to wrap your head around. However, I really like them, so here is a brief summary.
A surreal number $x$ consists of an ordered pair of sets written $\langle L_x\mid R_x\rangle$, where $L_x$ called the left set of $x$ and $R_x$ is called the right set. These sets both consist of other surreal numbers, with the requirement that if $x_l \in L_x$ and $x_r\in R_x$, then we have $x_l < x_r$. $x$ then signifies a surreal number between $L_x$ and $R_x$ (the first such number according to its generation, see below). Ordering is defined the following way: Given two surreal numbers $x = \langle L_x\mid R_x\rangle, y = \langle L_y\mid R_y\rangle$ we say that $x \leq y$ iff both the following are true:


*

*There is no $x_l\in L_x$ such that $y \leq x_l$

*There is no $y_r\in R_y$ such that $y_r\leq x$


(Note that in order to evaluate $y \leq x_l$ and $y_r\leq x$, you need to apply the same definition over again. This will, in practice, get very tedious for all but the simplest numbers. This recursive concept comes back when defining addition and multiplication.)
Actually, I wasn't being quite truthful earlier. A surreal number is an equivalence class of such pairs (this is what took me a long time to really appreciate). A pair itself is called a surreal number form. Two forms $x, y$ belong to the same equivalence class iff $x\leq y$ and $y\leq x$.
Each surreal number has a so-called "generation". The first surreal number (generation $0$) is $0 = \langle {}\mid{} \rangle$ where the left and right sets are empty. The next two surreal numbers (generation $1$) are $1 = \langle0\mid{}\rangle$ and $-1 = \langle{}\mid 0\rangle$. Generation $2$ consists of $-2 = \langle{}\mid -1\rangle$, $-\frac12 = \langle -1\mid 0\rangle$, $\frac12 = \langle 0\mid 1\rangle$ and $2 = \langle 1\mid{}\rangle$.
Here we can see the equivalence classes at work, because we also have $2 = \langle -1, 0, 1\mid {}\rangle$, and we have, for instance, $0 = \langle -2\mid 1\rangle$, because even though $-1$, $\frac12$ and $-\frac12$ are also between $-2$ and $1$, $0$ belongs to an earlier generation. You can check that we indeed have $\langle -2\mid 1\rangle\leq \langle {}\mid {}\rangle$ and at the same time $\langle {}\mid{}\rangle \leq \langle -2\mid 1\rangle$, while the same is not true if we swap $\langle {}\mid{}\rangle$ for $\langle 0\mid 1\rangle$.
We continue making finer and finer divisions in all the finite generations, every number that appear being a number of the form $\frac a{2^b}$, a dyadic fraction, until we get to the first infinite generation, $\omega$ (yes, the generations are ordinals), where all the real numbers suddenly pop up (for instance, $\sqrt 2 = \langle 1, \frac{5}{4}, \frac{11}{8}, \ldots{}\mid {}\ldots,\frac{12}{8}, \frac{3}{2}, 2\rangle$). We also get the first infinite ordinal, $\omega$ itself, as $\langle 1, 2, 3,\ldots{}\mid{}\rangle$, and its reciprocal $\frac1\omega = \langle {}\mid {}\ldots \frac18,\frac14,\frac12,1\rangle$.
So far I haven't talked about the arithmetic. Without that, there is no reason to call $\langle 0\mid 1\rangle$ $\frac12$ and not anything else. Given $x = \langle L_x\mid R_x\rangle$ and $y = \langle L_y\mid R_y\rangle$, addition is defined recursively by
$$
x + y = \langle \{x + y_l: y_l \in L_y\}\cup \{x_l + y:x_l\in L_x\} \mid \{x + y_r: y_r \in R_y\}\cup \{x_r + y:x_r\in R_x\}\rangle
$$
Multiplication is a bit more messy, so I'll use some shorthand:
$$
xy = \langle \{x_ly + xy_l - x_ly_l\}\cup\{x_ry + xy_r - x_ry_r\}\mid  \{x_ly + xy_r - x_ly_r\}\cup\{x_ry + xy_l - x_ry_l\}\rangle
$$
where subtraction is defined as one might expect, by negating the right number and adding. Negating is done by negating every element in the right and left sets, and swap the two around.
Just as with the ordinals, $\omega^2 = \omega\cdot \omega$ is a perfect square. However, here comes the fun part: Any positive surreal number has a square root. To get the square root of $\omega$, we need to so some more definitions (theoretically, one could, and should, justify every one of these names by performing the addition and multiplication to see that you get what you should, but that's a lot of work):
$$
\omega - 1 = \langle 1, 2, 3,\ldots\mid \omega\rangle\\
\omega - 2 = \langle 1, 2, 3,\ldots\mid \omega - 1\rangle\\
\omega - 3 = \langle 1, 2, 3,\ldots\mid \omega - 2\rangle\\
\vdots
$$
and then we get $\frac\omega2 = \langle 1, 2, 3,\ldots \mid\ldots, \omega - 3, \omega - 2, \omega - 1\rangle$. Similarily, we may define $\frac\omega2-1, \frac\omega2-2$, and so on, and we get $\frac\omega4 = \langle 1, 2, 3, \ldots\mid \ldots, \frac\omega2 - 3,\frac\omega2-2,\frac\omega2 - 1\rangle$. Then we may define $\frac\omega8, \frac\omega{16}$ and so on. Finally, we get $\sqrt\omega = \langle 1, 2, 3, 4,\ldots\mid \ldots,\frac\omega8,\frac\omega4,\frac\omega2,\omega\rangle$. We are now at generation $\omega^2$.
