Dependance of an SO(3) invariant function

I am looking to reduce the dependence of a function, knowing that it satisfies some invariance constraints. Let me first formulate my question by explaining the 2-dimensional case.

Imagine I have a function, that depends on two vectors in $$\mathbb{R}^2$$. Now suppose that I know this function is invariant under $$SO(2)$$ transformation on its vectors : $$f(R(\theta)\vec{r}_1, R(\theta)\vec{r}_2) = f(\vec{r}_1, \vec{r}_2)$$ where $$R(\theta)$$ is a rotation matrix.

The problem I am trying to solve is to reduce the dependence of the function to the minimal number of variables, which should be 3 instead of the 4 variables $$x_1, x_2, y_1, y_2$$.

Intuitively, I know that the transform : $$r_1 = \sqrt{x^2_1+y^2_1}\\ r_2 = \sqrt{x^2_2+y^2_2}\\ \phi_1 = \arctan(y_1/x_1) + \arctan(y_2/x_2) = \theta_1 + \theta_2\\ \phi_2 = \arctan(y_1/x_1) - \arctan(y_2/x_2) = \theta_1 - \theta_2$$ will be an answer because the group action takes $$\phi_1$$ to $$\phi_1+\theta$$ and leaves the rest unchanged, which means that $$f$$ is independent of $$\phi_1$$.

Question: How to solve exactly the same problem for 3 vectors in $$\mathbb{R}^3$$ and with $$H$$ invariant with respect to $$SO(3)$$.

Bonus: Is there a natural generalization to higher dimensions?

The transformation $$(x_1, y_1, x_2, y_2) \rightsquigarrow (r_1, r_2, \phi_1, \phi_2)$$ in the problem statement is only valid generically, i.e., on some open set, so I'll assume that a generically valid answer is sufficient for OP's purposes, but I'll make a comment at the end of the answer addressing what happens for nongeneric points.

We can treat all cases $$n$$ uniformly (but NB as $$n$$ grows the number of nongeneric cases to treat increases rapidly).

First, note that there is a bijective correspondence between:

1. functions $$f : (\Bbb R^n)^n \to Y$$ invariant under the action of $$SO(n)$$ and
2. functions $$\tilde f : SO(n) \backslash (\Bbb R^n)^n \to Y$$ (i.e., functions on the space of orbits),

characterized by $$f({\bf X}) = \tilde f(SO(n) \cdot {\bf X}),$$ where we've denoted $${\bf X} := ({\bf x}_1, \ldots, {\bf x}_n) \in (\Bbb R^n)^n .$$

A generic point $$\bf X$$ is a basis of $$\Bbb R^n$$, in which case the stabilizer $$G_{\bf X}$$ of $$\bf X$$ under the action of $$SO(n)$$ on $$(\Bbb R^n)^n$$ is trivial. (A point is generic in this sense if $$\det {\bf X} \neq 0$$, which defines an open, dense subset of $$(\Bbb R^n)^n$$). Thus the orbit of such an $$\bf X$$ has dimension $$\dim (SO(n) \cdot {\bf X}) = \dim (SO(n) / G_{\bf X}) = \dim SO(n) - \dim G_{\bf X} = \dim SO(n) = \frac{1}{2} n (n - 1) .$$

Since $$B$$ is open, a function on some neighborhood of $$SO(n) \cdot {\bf X}$$ in $$SO(n) \backslash (\Bbb R^n)^n$$---equivalently, an $$SO(n)$$-invariant function on some neighborhood of $${\bf X} \in (\Bbb R^n)^n$$---depends on $$\dim (\Bbb R^n)^n - \dim SO(n) = n^2 - \frac{1}{2} n (n - 1) = \frac{1}{2} (n + 1) n$$ variables.

We can realize this dependence explicitly: Any invertible square matrix $$\bf X$$ can be decomposed uniquely as a product $${\bf X} = Q R$$ of an orthogonal matrix $$Q$$ and an upper triangular matrix $$R$$ with positive diagonal entries (this is the QR decomposition). By allowing ourselves the possibility of negating the last column of $$Q$$ and the $$(n, n)$$-entry of $$R$$, we may assume that $$Q \in SO(n)$$. By definition an $$SO(n)$$-invariant function $$f$$ satisfies $$f({\bf X}) = f(QR) = f(R) ,$$ so by uniqueness we can identify $$SO(n)$$-invariant functions on a neighborhood of $$\bf X$$ (in fact, on all of $$B$$) with functions on an open subset of the space $$T(n, \Bbb R)$$ of upper triangular matrices, and in particular with functions of $$\dim T(n, \Bbb R) = \frac{1}{2} (n + 1) n$$ variables.

In principle we can use this factorization to write down explicitly how a set of $$\frac{1}{2} (n + 1) n$$ variables depends on the entries of $$\bf X$$.

Example In the case $$n = 2$$, the decomposition of $${\bf X} = \pmatrix{x_1 & x_2 \\ y_1 & y_2}$$ is $${\bf X} = Q \cdot \frac{1}{\sqrt{x_1^2 + y_1^2}} \pmatrix{x_1^2 + y_1^2 & x_1 x_2 + y_1 y_2 \\ 0 & x_1 y_2 - x_2 y_1}$$ for some $$Q \in SO(2)$$. So (at least on the open set $$\{x_1 y_2 - x_2 y_1 \neq 0\}$$) a $$SO(2)$$-invariant function $$f({\bf X}) = f(x_1, y_1, x_2, y_2)$$ depends only on \begin{align*} t_{11} &= \sqrt{x_1^2 + y_1^2} \\ t_{12} &= \frac{x_1 x_2 + y_1 y_2}{\sqrt{x_1^2 + y_1^2}} \\ t_{22} &= \frac{x_1 y_2 - x_2 y_1}{\sqrt{x_1^2 + y_1^2}} . \end{align*} These variables are related to the variables $$r_1, r_2, \phi_2$$ in the problem statement by $$r_1 = t_{11}, \qquad r_2 = \sqrt{t_{12}^2 + t_{22}^2}, \qquad \sin \phi_2 = \frac{t_{22}}{\sqrt{t_{12}^2 + t_{22}^2}} .$$ In fact, this decomposition works under the weaker genericity condition $${\bf x}_1 \neq {\bf 0}$$.

As an indication of what happens outside generic sets, consider the case $$n = 2$$ where $${\bf x}_1 = {\bf 0}$$. Then, there is a rotation $$A \in SO(2)$$ such that $$A {\bf x_2} = (||{\bf x}_2||, 0)$$, and so $$f({\bf 0}, {\bf x}_2) = f(A{\bf 0}, A{\bf x}_2) = f({\bf 0}, (||{\bf x}_2||, 0)) .$$ Thus, on the set $$\{{\bf 0}\} \times \Bbb R^2 \subset (\Bbb R^2)^2$$ an $$SO(2)$$-invariant function can be specified by a function $$g(a) := f({\bf 0}, (a, 0))$$ of $$1$$-variable, and we can specify an $$SO(2)$$-invariant function on $$(\Bbb R^2)^2$$ by:

• $$1$$ function of $$3$$ variables, and
• $$1$$ function of $$1$$ variable.