Is S a subgroup of GL(2, R)? $S = \left\{\begin{bmatrix} a & b \\ b & a \end{bmatrix} \in GL(2, R) \ \middle| \ a^2 + b^2 \neq 0\right\}$. S has the identity, also has the inverse  for every element. Also when $A, B \in S$, $AB \in S$. So shouldn't $S$ be a subgroup? (The answer is given it is not)
 A: Ok, let me try again to answer this question. First note that the condition that $a^2 + b^2 \neq 0$ doesn't mean much since all invertible matrices
$$
\pmatrix{a & b \\ b & a}
$$
will have $a^2 + b^2 \neq 0$. The only way that $a^2 + b^2 =0$ is if $a = b = 0$ and then the matrix clearly isn't invertible.
So $S$ really just consists of all matrices of the form
$$
\pmatrix{a & b \\ b & a}
$$
where $a^2 - b^2 \neq 0$.
You want to see if this is set $S$ is actually a subgroup. Since the product of two invertible matrices is invertible, so just check that the product of two matrices have the right form:
$$
\pmatrix{a & b \\ b & a}\pmatrix{c & d \\ d & c}
= \pmatrix{ac + bd & ad + bc \\ bc + ad & bd + ac}.
$$
You see that that the set is closed. The next observation is that 
$$
\pmatrix{1 & 0 \\ 0 & 1} \in S.
$$
Now we just have to show that for each matrix in $S$, the inverse is also in $S$. But we note that
$$
\pmatrix{a & b \\ b & a}^{-1} = \pmatrix{a/d & -b/d \\ -b/d & a/d}
$$
where $d = a^2 - b^2$ is the (non-zero) determinant and this is clearly in $S$ again.
In all you have checked (by definition) that $S$ is a (sub)group.
