Exercise on representations

I am stuck on an exercise in Serre, Abelian $\ell$-adic representations (first exercise of chapter 1).

Let $V$ be a vector space of dimension $2$, and $H$ a subgroup of $GL(V)$ such that $\det(1-h)=0$ for all $h \in H$.

Show that in some basis $H$ is a subgroup of either $\begin{pmatrix} 1 & * \\ 0 &* \\ \end{pmatrix}$ or $\begin{pmatrix} 1 & 0 \\ * &* \\ \end{pmatrix}$.

I know that this means that there is a subspace or a quotient of $V$ on which $H$ acts trivially, and I know it is enough to show that $V$ is not irreducible as representation of $H$, but I don't know how to do it.

-
Hint: Since that determinant is $0$, this means that $1 - h$ is not invertible, so it has a non-trivial kernel. Consider how $h$ acts on elements of that kernel. – Tobias Kildetoft Feb 26 '13 at 19:29
Can you be more precise ? Of course $h$ acts trivially on his $1$-eigen space, but it is not true that other elements of $H$ stabilize $\ker(1-h)$. – user10676 Mar 1 '13 at 15:18
Indeed, you need to split into two cases, depending on whether there is a non-zero element in the intersection of all the kernels of $1-h$ for $h\in H$. If there is not, you will need to do some more work in order to find some subspace where all the $h\in H$ act by scalars (I just realized that I don't really have a good idea how to do this). – Tobias Kildetoft Mar 1 '13 at 15:33

By hypothesis, for all $h \in H$ and basis $(e,f)$ one has $$\det(h(e)-e,h(f)-f) =0.$$

Let $g \in H$ different from the identity.

$\bullet$ Suppose that $1$ is the only eigen-value of $g$. Then in some basis $(e,f)$ the matrix of $g$ is $Mat(g) = \left( \begin{smallmatrix} 1&1\\0&1 \end{smallmatrix}\right)$. Let $h \in H$, then $$(1) \quad \det(h(e)-e,h(f)-f)=0,$$ $$(2) \quad \det(hg(e)-e,hg(f)-f)=\det(h(e)-e,h(e)+h(f)-f) = 0.$$ If $h(e)-e=0$, then $h(e)$ is colinear to $e$. If $h(f)-f$, then (2) shows that $h(e)$ is colinear to $e$. If $h(e) \neq e$ and $h(f)\neq f$, then $h(e)+h(f)-f$ is colinear to $h(e)-e$ (by (2)). The latter is colinear to $h(f)-f$ (by (1)). This implies that $h(e)$ is colinear to $e$. Hence $ke$ is fixed by $H$ and we are done.

$\bullet$ Suppose that $g$ has two distinct eigenvalues $1$ and $a$. Then is some basis $(e,f)$ : $Mat(g) = \left( \begin{smallmatrix} 1&0\\0&a \end{smallmatrix}\right)$.

Lemma: if $h \in H$ then (i) $h(e)=e$ or (ii) $h(f)$ is colinear to $f$.

Proof: We have $$\det(h(e)-e,h(f)-f) = 0,$$ $$\det(hg(e)-e,hg(f)-f) = \det(h(e)-e,a h(f)-f) = 0.$$ If $h(e) \neq e$, then $h(f)-f$ and $a h(f)-f$ are colinear to each other (because they are both colinear to $h(e)-e$). Since $a \neq 1$, this implies that $h(f)$ is colinear to $f$. QED

To conclude we have to show that either every $h \in H$ satisfies (i) or every $h \in H$ satisfies (ii). If not, then let $h \in H$ not satisfying (i) and $k \in H$ not satisfying (ii). The matrices of $h$ and $k$ have the forms $Mat(h) = \left( \begin{smallmatrix} 1&0\\\alpha&* \end{smallmatrix}\right)$ and $Mat(k) = \left( \begin{smallmatrix} 1&\beta\\0&* \end{smallmatrix}\right)$ with $\alpha \neq 0$ and $\beta \neq 0$. We check that $\det(hk-{\rm{Id}}) = -\alpha \beta$ which is a contradiction.

-