Is composition an operation on the set of all continuous mappings from R to R? After reading the same two pages in my algebra book over and over, all I've gotten is that an operation just has to be closed in the set under a mapping. 
I guess composition wouldn't be an operation if there was somehow a mapping that took it into the complex set. 
 A: Yes. This follows because if $f$ is continuous and $g$ is continuous then $f\circ g$ is continuous. One sees for instance from the sequential definition of continuity that if $x_n \rightarrow x$ that
$$\lim_{n \rightarrow \infty} f(g(x_n))=f\left(\lim_{n\rightarrow \infty} g(x_n)\right)=f\circ g \left(\lim_{n \rightarrow \infty} x_n\right)=f(g(x)).$$
A: By definition, a binary operation $*$ on a set $A$ is a mapping $ *:A\times A\to A$. Given $(a,b)\in A\times A$, it's usual to write $a*b$ to mean the "formally correct" $*(a,b)$.
Let's now consider your example and let $\mathcal{F}$ be the set of all continuous functions $\mathbb{R} \to \mathbb{R}$.
Could it be that the "correspondence" $\circ$ is in fact a function (or mapping)?
Let $(f,g)\in \mathcal{F}\times \mathcal{F}$. We wish to prove that $\circ (f,g)\in \mathcal{F}$, i.e., $f\circ g \in \mathcal{F}$.
Since $g$ maps to $\mathbb{R}$ and $f$ maps from $\mathbb{R}$ to $\mathbb{R}$ we know that $f\circ g$ maps from $\mathbb{R}$ to $\mathbb{R}$.
Therefore, the only thing left to guarantee that $f\circ g \in \mathcal{F}$, is to show that $f\circ g$ is continuous. That follows from real analysis theory or Jacob's answer.
From here, one can conclude that it's not possible for elements of $\mathcal{F}$ to somehow map to set not contained in $\mathbb{R}$. This means that your "somehow" possibility can't happen.
I interpreted "somehow a mapping that took it into the complex set" as "somehow a map that took it outside the real numbers", because, of course, $\mathbb{R} \subseteq \mathbb{C}$.
