The complex norm versus the Euclidean norm of a complex number Let $z\in \mathbb{C}$, defined as $z=a+ib$ where $a,b \in \mathbb{R}$.
The function $f:\mathbb{C}\to\mathbb{R}$:
$$
f(z)=z^*z=(a+ib)(a-ib)=a^2+b^2
$$
is invariant under the group $U(1)$.

Now take $\mathbf{z}$ to be a point on the complex plane. Thus, $\mathbf{z}\in \mathbb{R}\times \mathbb{R}$, defined as $\mathbf{z}=(a,b)$.
The function $g:\mathbb{R}\times \mathbb{R}\to \mathbb{R}$:
$$
g(\mathbf{z})=\mathbf{z}^T\mathbf{z}=\pmatrix{a&b}\pmatrix{a\\b}=a^2+b^2
$$
is invariant under the group $O(2)$.

Finally, take the function $h:\mathbb{R}\times\mathbb{R} \to \mathbb{R}$:
$$
h(a,b)=(a+ib)(a+ib)^*=(a+ib)(a-ib)=a^2+b^2
$$
is invariant under the group $U(1)$.

In the case of the function $g$ and $h$ how can I map to the set to the same end result, and yet end up with two different invariance groups?
(although I am a bit uneasy with $h$, as I am not sure how $U(1)$ can act on $\mathbb{R}\times\mathbb{R}$ --- so maybe someone can point out my misunderstanding. I am thinking the transformation is $h(a,b)\to h(Ua,Ub)\implies U^*U=I$)

edit:
I suppose that what I ought to investigate is $h(a,b)\to h(U(a,b))$, instead of $h(a,b)\to h(Ua,Ub)$? Otherwise $Ua,Ub$ is unable to capture the set of all possible transformations over $(a,b)$ that are possible under $U(a,b)$.
 A: First, your functions $g$ and $h$ have the same domain $\mathbb R \times \mathbb R$, the same range $\mathbb R$, and the same output $a^2 + b^2$ for each $(a,b)$ in the domain. Therefore, your functions $g$ and $h$ are the same function.
Next, as you suspect, it does not make sense to let $U(1)$ act on $\mathbb R \times \mathbb R$. You were right the first time, $U(1)$ acts on $\mathbb C$, it does not act on $\mathbb R \times \mathbb R$.
But you are missing an important point. Consider the standard bijection
$$I : \mathbb C \to \mathbb R \times \mathbb R
$$
defined by
$$I(a+bi)=(a,b)
$$
Consider also the isomorphism 
$$J : U(1) \to SO(2)
$$
defined by
$$J((c+si)) = \pmatrix{c & s \\ -s & c} \quad\text{for each $M=(c+si) \in U(1)$}
$$
I use the letters $c$ and $s$ because I am thinking in polar coordinates where $c+si = \cos(\theta) + \sin(\theta) \, i$ for some $\theta$. Also, I'm using $SO(2)$ instead of $O(2)$, the difference being that in $O(2)$ the determinant of the $2 \times 2$ matrix can be either $-1$ or $+1$, where $SO(2)$ is the subgroup of $O(2)$ for which the determinant equals $+1$.
Now there is a simple identity which is easy to check directly:
$$I(M(z))^T = J(M)(I(z)^T) \quad\text{for each $M = (c+si) \in U(1)$ and each $z=a+bi \in \mathbb C$}
$$
where I use the transpose operator $T$ to convert row vectors into a column vector, so that the row vector $I(z)=(a,b)$, after being transposed to a column vector, can be placed to the right of the $2 \times 2$ matrix $J(M) = \pmatrix{c & s \\ -s & c}$.
What this identity tells us intuitively is this. If we allow ourselves to blur the distinction between $\mathbb C$ and $\mathbb R \times \mathbb R$ -- which is very frequently done -- then the actions of the groups $U(1)$ and $SO(2)$ become similarly blurred. If we can tolerate this blurriness, we won't really lose anything. If we cannot tolerate this blurriness, then we can always fall back on the formalities of the bijection $I$ and the isomorphism $J$.
