Dependance of an SO(3) invariant function I am looking to reduce the dependence of a function, knowing that it satisfies some invariance constraints. Let me first formulate my question by explaining the 2-dimensional case.
Imagine I have a function, that depends on two vectors in $\mathbb{R}^2$. Now suppose that I know this function is invariant under $SO(2)$ transformation on its vectors :
$$
f(R(\theta)\vec{r}_1, R(\theta)\vec{r}_2) = f(\vec{r}_1, \vec{r}_2)
$$
where $R(\theta)$ is a rotation matrix.
The problem I am trying to solve is to reduce the dependence of the function to the minimal number of variables, which should be 3 instead of the 4 variables $x_1, x_2, y_1, y_2$.
Intuitively, I know that the transform :
$$
r_1 = \sqrt{x^2_1+y^2_1}\\
r_2 = \sqrt{x^2_2+y^2_2}\\
\phi_1 = \arctan(y_1/x_1) + \arctan(y_2/x_2) = \theta_1 + \theta_2\\
\phi_2 = \arctan(y_1/x_1) - \arctan(y_2/x_2) = \theta_1 - \theta_2
$$
will be an answer because the group action takes $\phi_1$ to $\phi_1+\theta$ and leaves the rest unchanged, which means that $f$ is independent of $\phi_1$.
Question: How to solve exactly the same problem for 3 vectors in $\mathbb{R}^3$ and with $H$ invariant with respect to $SO(3)$.
Bonus: Is there a natural generalization to higher dimensions?
 A: The transformation $(x_1, y_1, x_2, y_2) \rightsquigarrow (r_1, r_2, \phi_1, \phi_2)$ in the problem statement is only valid generically, i.e., on some open set, so I'll assume that a generically valid answer is sufficient for OP's purposes, but I'll make a comment at the end of the answer addressing what happens for nongeneric points.
We can treat all cases $n$ uniformly (but NB as $n$ grows the number of nongeneric cases to treat increases rapidly).
First, note that there is a bijective correspondence between:


*

*functions $f : (\Bbb R^n)^n \to Y$ invariant under the action of $SO(n)$ and

*functions $\tilde f : SO(n) \backslash (\Bbb R^n)^n \to Y$ (i.e., functions on the space of orbits),


characterized by $$f({\bf X}) = \tilde f(SO(n) \cdot {\bf X}), $$
where we've denoted $${\bf X} := ({\bf x}_1, \ldots, {\bf x}_n) \in (\Bbb R^n)^n .$$
A generic point $\bf X$ is a basis of $\Bbb R^n$, in which case the stabilizer $G_{\bf X}$ of $\bf X$ under the action of $SO(n)$ on $(\Bbb R^n)^n$ is trivial. (A point is generic in this sense if $\det {\bf X} \neq 0$, which defines an open, dense subset of $(\Bbb R^n)^n$). Thus the orbit of such an $\bf X$ has dimension
$$\dim (SO(n) \cdot {\bf X}) = \dim (SO(n) / G_{\bf X}) = \dim SO(n) - \dim G_{\bf X} = \dim SO(n) = \frac{1}{2} n (n - 1) .$$
Since $B$ is open, a function on some neighborhood of $SO(n) \cdot {\bf X}$ in $SO(n) \backslash (\Bbb R^n)^n$---equivalently, an $SO(n)$-invariant function on some neighborhood of ${\bf X} \in (\Bbb R^n)^n$---depends on
$$\dim (\Bbb R^n)^n - \dim SO(n) = n^2 - \frac{1}{2} n (n - 1) = \frac{1}{2} (n + 1) n$$
variables.
We can realize this dependence explicitly: Any invertible square matrix $\bf X$ can be decomposed uniquely as a product
$${\bf X} = Q R$$
of an orthogonal matrix $Q$ and an upper triangular matrix $R$ with positive diagonal entries (this is the QR decomposition). By allowing ourselves the possibility of negating the last column of $Q$ and the $(n, n)$-entry of $R$, we may assume that $Q \in SO(n)$. By definition an $SO(n)$-invariant function $f$ satisfies
$$f({\bf X}) = f(QR) = f(R) ,$$
so by uniqueness we can identify $SO(n)$-invariant functions on a neighborhood of $\bf X$ (in fact, on all of $B$) with functions on an open subset of the space $T(n, \Bbb R)$ of upper triangular matrices, and in particular with functions of $\dim T(n, \Bbb R) = \frac{1}{2} (n + 1) n$ variables.
In principle we can use this factorization to write down explicitly how a set of $\frac{1}{2} (n + 1) n$ variables depends on the entries of $\bf X$.
Example In the case $n = 2$, the decomposition of $${\bf X} = \pmatrix{x_1 & x_2 \\ y_1 & y_2}$$ is $${\bf X} = Q \cdot \frac{1}{\sqrt{x_1^2 + y_1^2}} \pmatrix{x_1^2 + y_1^2 & x_1 x_2 + y_1 y_2 \\ 0 & x_1 y_2 - x_2 y_1}$$
for some $Q \in SO(2)$. So (at least on the open set $\{x_1 y_2 - x_2 y_1 \neq 0\}$) a $SO(2)$-invariant function $f({\bf X}) = f(x_1, y_1, x_2, y_2)$ depends only on
\begin{align*}
 t_{11} &= \sqrt{x_1^2 + y_1^2} \\
 t_{12} &= \frac{x_1 x_2 + y_1 y_2}{\sqrt{x_1^2 + y_1^2}} \\
 t_{22} &= \frac{x_1 y_2 - x_2 y_1}{\sqrt{x_1^2 + y_1^2}} .
\end{align*}
These variables are related to the variables $r_1, r_2, \phi_2$ in the problem statement by
$$
    r_1 = t_{11}, \qquad
    r_2 = \sqrt{t_{12}^2 + t_{22}^2}, \qquad
    \sin \phi_2 = \frac{t_{22}}{\sqrt{t_{12}^2 + t_{22}^2}} .
$$
In fact, this decomposition works under the weaker genericity condition ${\bf x}_1 \neq {\bf 0}$.
As an indication of what happens outside generic sets, consider the case $n = 2$ where ${\bf x}_1 = {\bf 0}$. Then, there is a rotation $A \in SO(2)$ such that $A {\bf x_2} = (||{\bf x}_2||, 0)$, and so
$$f({\bf 0}, {\bf x}_2) = f(A{\bf 0}, A{\bf x}_2) = f({\bf 0}, (||{\bf x}_2||, 0)) .$$
Thus, on the set $\{{\bf 0}\} \times \Bbb R^2 \subset (\Bbb R^2)^2$ an $SO(2)$-invariant function can be specified by a function $g(a) := f({\bf 0}, (a, 0))$ of $1$-variable, and we can specify an $SO(2)$-invariant function on $(\Bbb R^2)^2$ by:


*

*$1$ function of $3$ variables, and

*$1$ function of $1$ variable.

