Sign up ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Suppose $G$ is a subgroup of $O_n(\mathbb{R})$ generated by finitely many reflections, where a reflection is defined to be a linear transformation fixing a hyperplane and sending a normal vector to the hyperplane to its negative. Suppose the normal vectors corresponding to these hyperplanes are linearly independent. Is G necessarily isomorphic to a Coxeter group?

This problem arose upon considering the fact that every Coxeter group is isomorphic to its Tits reflection representation, and I was wondering if there was a converse statement. Looking at the literature, a common statement is that this is true for finite reflection groups, but I could not find a statement for finitely generated reflection groups.

I might be missing some easy counterexample here as it seems like it would be a standard result if it were true.


share|cite|improve this question
It depends on what you mean by a Coxeter group. Take two lines in the Euclidean plane passing through the origin whose intersection angle is an irrational multiple of $2\pi$. The group generated by reflection in these two lines is an infinite dihedral group, and the restriction of its action to the unit circle has dense orbits in the circle. This is not a Coxeter group in the classical sense of a discrete reflection group. However it is an abstract Coxeter group. Which of these two senses are you interested in? – Lee Mosher May 22 '12 at 13:09

2 Answers 2

No. For a specific counterexample, let $p$ be the root of $$p^3-3p^2+1$$ which is roughly $0.652704$ and chose three vectors in $\mathbb{R}^3$ at angles $\cos^{-1}(p)$ with respect to each other. Let $a$, $b$ and $c$ be the reflections in these vectors. Then $\langle a,b \rangle$, $\langle a,c \rangle$ and $\langle b,c \rangle$ are infinite dihedral groups, but $(abc)^6=\mathrm{Id}$. So this is not a Coxeter group with those generators, and I highly doubt it is a Coxeter group at all.

OK, so how did I find this? I started out looking for reflections $a$, $b$ and $c$ such that the dihedral groups $\langle a,b \rangle$, $\langle a,c \rangle$, $\langle b,c \rangle$ are infinite but $abc$ had finite order. I knew that a good way to compute with vectors in $\mathbb{R}^3$ was in terms of their matrix of inner products. Let $p = \langle v_a, v_b \rangle$, $q = \langle v_a, v_c \rangle$, $r = \langle v_b, v_c \rangle$ where $s$ is the reflection in unit vector $v_s$. I claim that $a$, $b$, $c$ will be a counter-example as long as $(p,q,r)$ are real numbers in the interval $(-1,1)$ such that
(1) $\frac{1}{\pi} \cos^{-1} p$, $\frac{1}{\pi} \cos^{-1} q$, $\frac{1}{\pi} \cos^{-1} r$ are irrational
(2) $1+2pqr > p^2+q^2+r^2$ and
(3) $\frac{1}{\pi} \cos^{-1} \left( \frac{p^2+q^2+r^2-pqr-2}{2} \right)$ is rational.

Proof: First we show that the vectors exist. There is a collection of matrices with the given inner products if and only if the Gram matrix $$\begin{pmatrix} 1 & p & q \\ p & 1 & r \\ q & r & 1 \\ \end{pmatrix}$$ is positive definite. By Sylvester's criterion (since $p \in (-1,1)$), this will occur if and only if the determinant of the Gram matrix is positive. That determinant is $1+2pqr - p^2-q^2-r^2$, explaining condition (2).

The pairs $(a,b)$, $(a,c)$ and $(b,c)$ will generate infinite dihedral groups as long as $\frac{1}{\pi} \cos^{-1}(p)$, $\frac{1}{\pi} \cos^{-1}(q)$ and $\frac{1}{\pi} \cos^{-1}(r)$ are all irrational. That explains condition (1).

I had Mathematica compute the characteristic polynomial of $abc$. It is $$(x+1) (x^2 - (p^2+q^2+r^2 - pqr -2) x + 1).$$ (We knew that the $-1$ root would be there, since it $abc$ is an orientation reversing orthogonal map.) So $abc$ will have finite order if and only if $\frac{1}{\pi} \cos^{-1} \left( \frac{p^2+q^2+r^2-pqr-2}{2} \right)$ is rational, explaining condition (3).

At this point, you should have the strong feeling that you are already done. Fixing some candidate value $2 \cos (\frac{m}{n} \pi)$ for $p^2+q^2+r^2-pqr-2$, there should be a whole surface of solutions to (3). Then (2) will cut out an open subset of that surface, and (1) will cut out a dense subset of that. So there should be such points.

Looking for an explicit solution to give you, I decided to set $p=q=r$. (This was motivated by ease of computation, not by any deep theory.) Condition (2) then becomes $p \in (-1/2, 1)$. Numeric computation shows that $p^2+q^2+r^2-pqr-2$ is then in $[-2,0)$; I decided to set it equal to $-1 = 2 \cos (2 \pi/3)$ to enforce condition (3). So my $p$ is one of the roots of $$3 p^2 - p^3 - 2 = -1$$ which lies in the desired range.

The roots of this equation are $(-0.532089, 0.652704, 2.87939)$, so the root at $0.652704$ is valid for my purposes. Note that these roots provide an immediate proof that $\frac{1}{\pi} \cos^{-1}(p)$ is not rational: If $p$ were in $\cos (\pi \mathbb{Q})$, then so would all its Galois conjugates be, and this polynomial would not have a root outside $(-1,1)$.

share|cite|improve this answer

In a different direction, you might be interested in the following result of Deodhar (see also Dyer). Let $W$ be any Coxeter group, possibly infinite, or even of infinite rank. (Rank is the number of Coxeter generators.) Let $W'$ be any subgroup of $W$ generated by some set of reflections of $W$. Then $W'$ is a Coxeter group.

Warning 1: It can happen that the rank of $W'$ is larger than that of $W$, and even that $W$ has finite rank and $W'$ has infinite rank.

Warning 2: If $V$ and $V'$ are the reflection representations of $W$ and $W'$, then there is a natural $W'$-equivariant map $V' \to V$, but this map need be neither injective nor surjective.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.