# Does this set have any non-trivial rotational symmetries?

Let $\mathbf{v}:=(a,b,c,d)^T\in\mathbb{R}^4.$ Then does the set of points in $\mathbb{R}^4$ $$S:=\{a,b,c,d\;|\;ab>cd \;\wedge\; a+b<0\}$$ have any non-trivial rotational symmetries about the origin, where non-trivial means that the rotations are not of order $1$ (identity) or $2$ (reflections)? I.e., does there exist any orthogonal $4\times 4$ matrices $\mathbf{M}$ such that if $\mathbf{v}\in S$ then $\mathbf{Mv}\in S$?

I suspect that there does not, but how can I approach this? Any ideas and/or observations are welcome.

Thanks.

Edit: Note that $\mathbf{v}$ can be rewritten as a matrix $\mathbf{V}:=\pmatrix{a & c \\ d & b}$ such that $S$ is defined as $$S:=\{\mathbf{A}\;|\;\det\mathbf{A}>0\;\wedge \; \mathrm{tr}\,\mathbf{A}<0\},$$ but then the rotation matrix becomes more obscure.

If $M$ maps $S$ into $S$, it must map its closure $\overline{S}=\{(a,b,c,d): ab\ge cd,\ a+b\le0\}$ into $\overline{S}$. Note that a one-dimensional linear subspace is contained in $\overline{S}$ if and only if it is the linear span of a vector in one of the two sets below: \begin{align*} X&=\{(0,0,c,d):\ (c,d)\ne(0,0),\ cd\le0\},\\ Y&=\{(a,-a,c,d):\ a\ne0,\, -a^2\ge cd\}. \end{align*} Since $M$ preserves one-dimensional linear subspaces in $\overline{S}$, it must map $X\cup Y$ into $X\cup Y$.

Vectors in $X$ and $Y$ differ in a crucial way. It is not hard to show that:

• (a) if $x\in X$ is normal to another vector $x'\in X\cup Y$, then $x'$ must also belong to $X$ and both $x,x'$ are scalar multiples of $\pm e_3$ or $\pm e_4$;

• (b) if $y\in Y$ is normal to another vector $y'\in X\cup Y$, then $y$ must be a scalar multiple of $(1,-1,c,-\frac 1c)$ for some $c\ne0$ and $y'$ must be a scalar multiple of $(1,-1,-\frac 1c,c)$.

Therefore, every pair of orthonormal vectors $(y,y')\in Y^2$ is a limit point in $(X\cup Y)^2$, while every pair of orthonormal vectors $(x,x')\in X^2$ is isolated. Since $M$ preserves pairs of orthonormal vectors as well as converging sequences, it must preserve pairs of orthonormal vectors in $X$, and also pairs of orthonormal vectors in $Y$.

Hence (a) implies that the unordered set $\{Me_3,Me_4\}$ is equal to $\pm\{e_3,\pm e_4\}$. As $M(e_3-e_4)$ must reside in $\overline{S}$, its value cannot be $\pm(e_3+e_4)$. Hence $(Me_3,Me_4)=\pm(e_3,e_4)$ or $\pm(e_4,e_3)$ and in turn $M$ must map $X$ onto $X$.

By (b), there are only two pairs of mutually orthogonal vectors $y,y'\in Y$ that satisfy $\|y\|=\|y'\|=2$ and $\langle y,e_3\rangle=-\langle y,e_4\rangle=1$, namely, $y=(a,-a,1,-1)$ and $y'=(a,-a,-1,1)$ where $a=\pm1$. Since $(Me_3,Me_4)=\pm(e_3,e_4)$ and $M$ preserves norms, inner products and mutually orthogonal vectors of $Y$, we see that $My=\pm(1,-1,0,0)+M(e_3-e_4)$ and hence $M(e_1-e_2)=\pm(e_1-e_2)$.

But then we get $M(e_1+e_2)=\pm(e_1+e_2)$, because $M$ is an isometry and $e_1+e_2$ is orthogonal to $\operatorname{span}\{e_1-e_2,e_3,e_4\}$, which is an invariant subspace of $M$.

Using the fact that $M$ is an isometry, $M(e_1-e_2)=\pm(e_1-e_2)$ and $M(e_1+e_2)=\pm(e_1+e_2)$, one can easily infer that $(Me_1,Me_2)=(e_1,e_2),(-e_1,-e_2),(e_2,e_1)$ or $(-e_2,-e_1)$. As $M$ also maps $S$ into $S$ (and preserves quadruples $(a,b,c,d)$ such that $a+b<0$), we conclude that $(Me_1,Me_2)$ can only be $(e_1,e_2)$ or $(e_2,e_1)$.

Consequently, $M=\pmatrix{M_1&0\\ 0&M_2}$, where $M_1\in\{I,R\},\ M_2\in\{I,-I,R,-R\}$ and $R=\pmatrix{0&1\\ 1&0}$. In other words, the answer to your question is negative.

• This is a (seemingly) beautiful proof that I will need some time to digest (as its methods are totally new to me). Two things: a) If I accept the answer now, may I come back at some later time and ask a couple of questions regarding the proof, if needed? b) Where can I get more exposure to this kind of thing? I'm very interested in methods to determine symmetry groups of sets such as $S,$ which aren't finite polygons or the like. Thank you. Feb 25 '17 at 18:39
• @Lovsovs (a) You don't need to accept any proof, especially when you don't understand it. Not to say that my proof may contain errors, and other users may come up with better proofs. In short, whether to and when to accept an answer is straightly your call. (b) I didn't view this as a group theoretic problem, so I don't think I can give you any reference in this regard. Feb 25 '17 at 19:14
• @Lovsovs Anyway, I viewed the problem as a linear preserver problem. It is a customary trick in such a problem to identify some sets with nice structures that are invariant under the linear transformation in question. In your case, those nice sets are firstly $X$ and $Y$, and then the pairs of orthonormal vectors. We use these sets with nice geometric or topological properties to narrow down the possible choices of $M$. This was also explained in an answer of mine in another thread. Feb 25 '17 at 19:22
• Thank you, that was very helpful, will look into it! Feb 25 '17 at 19:47