General approach to determining if a subset is a subgroup if it has finite order I am a little confused as how to approach problems that ask whether a subset is a subgroup given that it has the property of being of finite order e.g. in the case for $GL(N,\mathbb{R}$). What property of a subset does being finite violate exactly?
EDIT: Sorry for the ambiguity, I mean take for example a subset $H$ of the group $G=GL(N,\mathbb{R}$). If $H$ is the set of elements of $G$ of finite order, why exactly is $H$ not a subgroup?
 A: There are finite subgroup of $GL(N, \mathbb{R})$.
A set of elements of finite order, $H \subset GL(N, \mathbb{R})$, may or may not be a subgroup.
$H$ is a subgroup if it contains the identity element of $GL(N, \mathbb{R}) = I_N$, and if it is closed under matrix multiplication (for $A, B \in H$ it is also true that $AB$ and $BA \in H$), and if for every element $h\in H, \; h^{-1} \in H$.
If all these properties are satisfied, then $H$, finite or otherwise, is a subgroup of $GL(N, \mathbb{R})$.  If any of these properties fail to hold, then $H$ is strictly a subset of $GL(N, \mathbb{R})$.

Challenge One can also prove a stronger statement:

If $H$ is a non-empty finite subset of a group $G$, then $H$ is a subgroup of $G$ if and only if it is closed under the binary operation of $G$. 

That is, if $H$ is a non-empty finite subset of group $G$, then it suffices to prove that it is closed under the group operation of $G$. To show this, one shows that closure of non-empty finite $H$ ensures that the identity of $G$ is necessarily in $H$, and for every element $h\in H$, $h^{-1} \in H.$
A: In your particular case, consider the two elements of $\text{GL}(2, \mathbf{R})$
\begin{equation}
a = \begin{bmatrix}-1 & 0\\0 & 1\end{bmatrix}, \qquad
b = \begin{bmatrix}-1 & -1\\0 & 1\end{bmatrix}.
\end{equation}
Verify that $a^2 = b^2 = 1 = \begin{bmatrix}1 & 0\\0 & 1\end{bmatrix}$. 
Now consider
\begin{equation}
c = a b = \begin{bmatrix}1 & 1\\0 & 1\end{bmatrix},
\end{equation}
and note that 
\begin{equation}
c^n = \begin{bmatrix}1 & n\\0 & 1\end{bmatrix},
\end{equation}
so that $c$ has infinite order.
