Is a bijective, norm-preserving and rotation-invariant mapping linear? Let $h: \mathbb{S}^N\to \mathbb{S}^N$, where $\mathbb{S}^N$ is the $N$-dimensional unit hypersphere in $\mathbb{R}^{N+1}$, be a mapping with the following properties:

*

*$h$ is bijective and smooth.

*$\left\|h(x)\right\| = 1$ (given that we operate on a hypersphere).

*$h(x)^\top h(y) = h(Rx)^\top h(Ry)$ for any rotation $R$, i.e. the inner product is invariant to rotations in the domain of $h$.

Is $h$ a linear or affine transformation (i.e. is $h(x) = Qx$ where $Q$ is a rotation)?

It took me a few days, but I think I finally found a proof using that if $h(x)^\top h(y) = x^\top y$ for all $x, y$ then $h$ is linear. This can be shown by realising that if $h(x)^\top h(y) \ne x^\top y$ for any $x, y$ then $h$ cannot be bijective, see the proof below.
 A: Here is an initial step: Every such map must be a conformal diffeomorphism.
To see it consider a point $z\in S^N$ and let $R_z$ be a rotation taking $h(z)$ to $z$. We look at $\tilde h:= R_z\circ h$ to note that
$$(\tilde h(Rx) , \tilde h(Ry) ) =(R_z h(Rx), R_z h(Ry)) = (h(x), h(y)) = (R_z h(x), R_z h(y))$$
and see that this enjoys the same property as $h$ with the added benefit that $z$ is fixed point of $\tilde h$. Now let $x(t)$ and $y(s)$ be two differentiable paths with $x(0)=y(0)=z$. Then for an arbitrary rotation $R$ with $R(z)=z$ one has:
$$( D_z\tilde h ( R x'(0)) , D_z\tilde h(R y'(0) ) )= \frac{d}{dt}\frac d{ds}(\tilde h(R x(t)), \tilde h(Ry(t) )\lvert_{s,t=0}=  \frac{d}{dt}\frac d{ds}(\tilde h( x(t)), \tilde h(y(t) )\lvert_{s,t=0}\\  
=( D_z\tilde h ( x'(0)) , D_z\tilde h( y'(0) ) ).$$
Since this then holds for all tangent vectors $x'(0), y'(0)$ one has that $R^T(D_z\tilde h)^T D_z\tilde h R = (D_z\tilde h)^T D_z \tilde h$ for any rotation preserving the point $z$. This implies that $(D_z\tilde h)^T D_z\tilde h$ is a scalar multiple of the identity on $T_zS^N$, ie that $D_z\tilde h = \lambda S$ for some $\lambda\in \Bbb R$ and a rotation $S$ of $T_zS^N$.
This means that $D_z\tilde h$ is a conformal mapping $T_zS^N\to T_zS^N$. Since $R_z$ is an isometry one has that $D_zh = D_z(R_z^{-1}\tilde h)$ is then also a conformal mapping. Since $z$ was arbitrary $h$ is a conformal map.
However it is not true that every conformal diffeomorphism of the sphere satisfies the desired equation, it is easy to construct counter examples on $S^2$ via Möbius transformations. This leads me to believe that only the isometries, ie the rotations, have this property.
A: Here is an attempt for a formal proof using Theorem 1 in [1] which basically states that $h$ is linear if $h(x)^\top h(y) = x^\top y$. We proof by contradiction that $h$ has this property. Let $\alpha(x,y) = \cos^{-1}(x^\top y)$ be the angular distance between $x$ and $y$. The proof has four main steps:

*

*If $\exists x, y\,\,\alpha(h(x), h(y)) > \alpha(x, y)$, then there exists a $z$ on the geodesic between $x$ and $y$ such that $\alpha(h(x), h(z)) > \alpha(x, z) < \epsilon$ for a given $\epsilon > 0$.


*If $\exists x, y\,\,\alpha(h(x), h(y)) > \alpha(x, y)$ then there exists an $h'$ fulfilling all desired properties (1-3) such that $\alpha(h'(x), h'(y)) < \alpha(x, y)$ (and vice versa).


*If $\exists x, y\,\,\alpha(h(x), h(y)) < \alpha(x, y)$ then $\alpha(h(x), h(z)) < \alpha(x, z)$ for all $z$ with $\alpha(x, y) = \alpha(x, z)$.


*If $\alpha(h(x), h(y)) < \alpha(x, y) < \epsilon$ for any $\epsilon > 0$ then $h$ cannot be bijective, contradicting the first assumption on $h$.
This concludes the proof. I now prove each part individually:
Proof of part 1
Assume there exists $x, y$ s.t. $\alpha(h(x), h(y)) > \alpha(x, y)$. Now consider a point $z$ and a rotation $R$ such that $z = Rx$ and $y = Rz$ (i.e. $z$ is a midpoint of $x$ and $y$ on the hypersphere). Then
\begin{align}
  2\alpha(x, z) &= \alpha(x, y), \\
                &< \alpha(h(x), h(y)), \\
                &\leq \alpha(h(x), h(z)) + \alpha(h(z), h(y)) , \\
                &= \alpha(h(x), h(z)) + \alpha(h(Rx), h(Rz)), \\
                &= 2\alpha(h(x), h(z))
\end{align}
where I used that the angular distance is a metric (and thus I can use the triangle inequality). Hence, $\alpha(x, z) < \alpha(h(x), h(z))$. We now just repeat this argument several times (each time halving the angle between $x$ and $z$ until $\alpha(x, z) < \epsilon$).
Proof of part 2
For any given $h$ fulfilling assumptions (1-3), its inverse $h^{-1}$ also fulfills (1-3). If there exists $x, y$ such that $\alpha(h(x), h(y)) > \alpha(x, y)$, then $\alpha(h^{-1}(x'), h^{-1}(y')) < \alpha(x', y')$ for $x' = h(x)$ and $y' = h(y)$.
Proof of part 3
For any $x^\top y = x^\top z$ we can find a rotation $R$ such that $x = Rx$ (i.e. $x$ is a fixed point of $R$) and $y = Rz$. Then if $\alpha(h(x), h(y)) < \alpha(x, y)$ we find that
$$\alpha(h(x), h(z)) = \alpha(h(Rx), h(Rz)) = \alpha(h(x), h(y)) < \alpha(x, y).$$
Proof of part 4
Given part 1., 2. and 3. we know that if there exists any $h$ for which there exists $x, y$ such that $h(x)^\top h(y) \ne x^\top y$, there must be a $\tilde h$ s.t. $\alpha(\tilde h(x), \tilde h(y)) < \alpha(x, y) < \epsilon$ for all $z$ with $x^\top y = x^\top z$. Choose any $x', y'$ such that $\tilde h(x') = -\tilde h(y')$ (which must exist because $\tilde h$ is bijective). Let further $z'$ be on the geodesic between $x'$ and $y'$ such that $\alpha(\tilde h(x'), \tilde h(z')) < \alpha(x', z') < \epsilon$ and let $N$ be such that $N\alpha(x', z') = \alpha(x', y')$ (this can be made precise in the limit $\epsilon\to 0$ and $N\to\infty$). Then,
\begin{align}
   \pi &= \alpha(\tilde h(x'), -\tilde h(x')), \\
       &= \alpha(\tilde h(x'), \tilde h(y')), \\
       &\leq N \alpha(\tilde h(x'), \tilde h(z')), \\
       &< N\alpha(x', z'), \\
       &= \alpha(x', y'), \\
       &\leq \pi.
\end{align}
This contradiction completes the proof.
[1] Linear mappings approximately preserving orthogonality, https://reader.elsevier.com/reader/sd/pii/S0022247X04007474?token=8EDC87750F4B5D379A98264CBF42E0CC2F4B73DFF6217BE5CB03B27A4ABD495E189DF508816E4E3DCC3E9ACEC5DA5B73
A: Given any orthogonal linear transformation $R$, let $R'=hRh^{-1}$.  One then has for every $x$ and $y$ in $\mathbb{S}^N$ that
$$
  ⟨R'(x),R'(y)⟩ =
  ⟨hRh^{-1}(x),hRh^{-1}(y)⟩ =
  ⟨hh^{-1}(x),hh^{-1}(y)⟩ =
  ⟨x, y⟩,
  $$
so $R'$ preserves inner-producs and hence
$R'$ must be an orthogonal transformation
(Theorem 1 in reference (1) cited in @w382903's answer).
The map
$$
  \phi: R∈O(n) \mapsto hRh^{-1}\in O(n)
  $$
is therefore an automorphism of the orthogonal group $O(n)$.  Since $\phi$ is continuous it
must preserve the connected component of the identity and hence $\phi$ restricts to an automorphism of $SO(n)$.
Since all automorphisms of $SO(n)$ are given by conjugation by an orthogonal transformation (see this post), there exists $U$ in $O(n)$
such that
$$
  hRh^{-1} = URU^{-1},
  $$
for all $R$ in $SO(n)$.  This translates to $U^{-1} hR = RU^{-1} h$, which means that
$$
  k:= U^{-1} h
  $$
commutes with every $R$
in $SO(n)$.
(We can't yet conclude that $k$ lies in the center of any matrix group in sight because we don't yet know it is
linear, but we are pretty close!)
Let us next prove that for every $x$ in $\mathbb{S}^N$, one has that $k(x)=\pm x$.  Assuming by contradiction that this
is not so, one may find a rotation $R$ such that $R(x)=x$, and $R(k(x))\neq k(x)$ (this requires that $N\geq 3$, which we
assume from now on).  Then
$$
  R(k(x)) =   k(R(x)) = k(x),
  $$
a contradiction.
This proves the claim and since $k$ is continuous, the choice of sign "$\pm$" must be constant for every $x$, meaning that $k$ is either the identity $I$ or
$-I$, and consequently
$$
  h = Uk = \pm U,
  $$
as desired.
