# Is every finitely generated reflection group a Coxeter group?

Suppose $G$ is a subgroup of $O_n(\mathbb{R})$ generated by finitely many reflections, where a reflection is defined to be a linear transformation fixing a hyperplane and sending a normal vector to the hyperplane to its negative. Suppose the normal vectors corresponding to these hyperplanes are linearly independent. Is G necessarily isomorphic to a Coxeter group?

This problem arose upon considering the fact that every Coxeter group is isomorphic to its Tits reflection representation, and I was wondering if there was a converse statement. Looking at the literature, a common statement is that this is true for finite reflection groups, but I could not find a statement for finitely generated reflection groups.

I might be missing some easy counterexample here as it seems like it would be a standard result if it were true.

Thanks!

-
It depends on what you mean by a Coxeter group. Take two lines in the Euclidean plane passing through the origin whose intersection angle is an irrational multiple of $2\pi$. The group generated by reflection in these two lines is an infinite dihedral group, and the restriction of its action to the unit circle has dense orbits in the circle. This is not a Coxeter group in the classical sense of a discrete reflection group. However it is an abstract Coxeter group. Which of these two senses are you interested in? – Lee Mosher May 22 '12 at 13:09

No. For a specific counterexample, let $p$ be the root of $$p^3-3p^2+1$$ which is roughly $0.652704$ and chose three vectors in $\mathbb{R}^3$ at angles $\cos^{-1}(p) \approx 0.273635 \pi$ with respect to each other. (Since $0.273635<1/2$, the angle is acute and this is clearly possible.) Let $a$, $b$ and $c$ be the reflections in these vectors. Then $\langle a,b \rangle$, $\langle a,c \rangle$ and $\langle b,c \rangle$ are infinite dihedral groups, but $(abc)^6=\mathrm{Id}$. So this is not a Coxeter group with those generators, and I highly doubt it is a Coxeter group at all.

OK, so how did I find this? I started out looking for reflections $a$, $b$ and $c$ such that the dihedral groups $\langle a,b \rangle$, $\langle a,c \rangle$, $\langle b,c \rangle$ are infinite but $abc$ had finite order. I knew that a good way to compute with vectors in $\mathbb{R}^3$ was in terms of their matrix of inner products. Let $p = \langle v_a, v_b \rangle$, $q = \langle v_a, v_c \rangle$, $r = \langle v_b, v_c \rangle$ where $s$ is the reflection in unit vector $v_s$. I claim that $a$, $b$, $c$ will be a counter-example as long as $(p,q,r)$ are real numbers in the interval $(-1,1)$ such that
(1) $\frac{1}{\pi} \cos^{-1} p$, $\frac{1}{\pi} \cos^{-1} q$, $\frac{1}{\pi} \cos^{-1} r$ are irrational
(2) $1+2pqr > p^2+q^2+r^2$ and
(3) $\frac{1}{\pi} \cos^{-1} \left( \frac{p^2+q^2+r^2-pqr-2}{2} \right)$ is rational.

Proof: First we show that the vectors exist. There is a collection of matrices with the given inner products if and only if the Gram matrix $$\begin{pmatrix} 1 & p & q \\ p & 1 & r \\ q & r & 1 \\ \end{pmatrix}$$ is positive definite. By Sylvester's criterion (since $p \in (-1,1)$), this will occur if and only if the determinant of the Gram matrix is positive. That determinant is $1+2pqr - p^2-q^2-r^2$, explaining condition (2).

The pairs $(a,b)$, $(a,c)$ and $(b,c)$ will generate infinite dihedral groups as long as $\frac{1}{\pi} \cos^{-1}(p)$, $\frac{1}{\pi} \cos^{-1}(q)$ and $\frac{1}{\pi} \cos^{-1}(r)$ are all irrational. That explains condition (1).

I had Mathematica compute the characteristic polynomial of $abc$. It is $$(x+1) (x^2 - (p^2+q^2+r^2 - pqr -2) x + 1).$$ (We knew that the $-1$ root would be there, since it $abc$ is an orientation reversing orthogonal map.) So $abc$ will have finite order if and only if $\frac{1}{\pi} \cos^{-1} \left( \frac{p^2+q^2+r^2-pqr-2}{2} \right)$ is rational, explaining condition (3).

At this point, you should have the strong feeling that you are already done. Fixing some candidate value $2 \cos (\frac{m}{n} \pi)$ for $p^2+q^2+r^2-pqr-2$, there should be a whole surface of solutions to (3). Then (2) will cut out an open subset of that surface, and (1) will cut out a dense subset of that. So there should be such points.

Looking for an explicit solution to give you, I decided to set $p=q=r$. (This was motivated by ease of computation, not by any deep theory.) Condition (2) then becomes $p \in (-1/2, 1)$. Numeric computation shows that $p^2+q^2+r^2-pqr-2$ is then in $[-2,0)$; I decided to set it equal to $-1 = 2 \cos (2 \pi/3)$ to enforce condition (3). So my $p$ is one of the roots of $$3 p^2 - p^3 - 2 = -1$$ which lies in the desired range.

The roots of this equation are $(-0.532089, 0.652704, 2.87939)$, so the root at $0.652704$ is valid for my purposes. Note that these roots provide an immediate proof that $\frac{1}{\pi} \cos^{-1}(p)$ is not rational: If $p$ were in $\cos (\pi \mathbb{Q})$, then so would all its Galois conjugates be, and this polynomial would not have a root outside $(-1,1)$.

-
I am a bit confused. Isn't an abstract Coxeter group defined to be one generated by reflections? So why is this not a Coxeter group? – Vishal Gupta Dec 10 '15 at 15:10
@VishalGupta An abstract Coxter group is defined to have generators $s_1$, $s_2$, ..., $s_n$ and relations $s_i^2=e$ and $(s_i s_j)^{m_{ij}}=e$ for some $2 \leq m_{ij} \leq \infty$. Nontrivial theorems: Any finite group generated by reflections is a Coxeter group, and any Coxeter group has a faithful rep where the $s_i$ act by reflections. But an infinite group generated by reflections can have relations of other forms, as this example shows. – David Speyer Dec 10 '15 at 17:10
But how do we know that a relation of the type $(abc)^6 = 1$ not implied by the axioms on the orders of $s_{i}s_{j}$. – Vishal Gupta Dec 10 '15 at 18:02
No, in the second I am right, although the reflections are do not preserve a positive definite quadratic form. Define a vector space $V$ with basis $\alpha_i$ and a symmetric bilinear form on $V$ by $\langle \alpha_i, \alpha_i \rangle = 2$ and $\langle \alpha_i, \alpha_j \rangle = -2 \cos(\pi/m_{ij})$ for $i \neq j$. Then the group generated by the reflections $x \mapsto x - \langle \alpha_i, x \rangle \alpha_i$ is isomorphic to the Coxeter group. The group will be finite if and only if the quad. form is positive definite. – David Speyer Dec 10 '15 at 18:08
Regarding the first: We have $m_{12}=m_{13}=m_{23}=\infty$, so the only relations are $a^2=b^2=c^2=1$. It is easy to give groups which obey these relations but don't obey $(abc)^6=1$ -- for example, use three generic reflections in $3$-space. – David Speyer Dec 10 '15 at 18:09

In a different direction, you might be interested in the following result of Deodhar (see also Dyer). Let $W$ be any Coxeter group, possibly infinite, or even of infinite rank. (Rank is the number of Coxeter generators.) Let $W'$ be any subgroup of $W$ generated by some set of reflections of $W$. Then $W'$ is a Coxeter group.

Warning 1: It can happen that the rank of $W'$ is larger than that of $W$, and even that $W$ has finite rank and $W'$ has infinite rank.

Warning 2: If $V$ and $V'$ are the reflection representations of $W$ and $W'$, then there is a natural $W'$-equivariant map $V' \to V$, but this map need be neither injective nor surjective.

-