Let $G$ denote the groups of $n\times n$ invertible matrices and $H$ be the subgroup of invertible upper triangular matrices. For $n=2$, by row reduction, or equivalently LU decomposition, it is straightforward to see that $$H\backslash G/H=\{H, HpH\}$$ where $p$ is the $2\times 2$ permutation matrix different from $I$. A similar argument shows that for $n>2$ the double coset $H\backslash G/H$ is a finite set, but I am curious to know if there is an algebraic/combinatorial way to count the number of its elements. It is also quite probable that this has already been worked out because of Bruhat decomposition. Any thoughts are appreciated!
1 Answer
This is a CW answer to remove this question from the unanswered list, as it was answered in comments by Jyrki Lahtonen: There are $n!$ double cosets, and representatives for them are the permutation matrices. This result is known as Bruhat Decomposition.