Any element $g$ of $GL(2,p)$ of order $p$, $p$ prime, is conjugate to $\begin{bmatrix}1&1\\0&1\end{bmatrix}$ 
Any element $g$ of $GL(2,p)$ of order $p$, $p$ prime, is conjugate to
  $\begin{bmatrix}1&1\\0&1\end{bmatrix}.$

I showed that $\langle g\rangle $ acts on the set $X$ of vectors with entries in $ F_p$ and hence that $g$ fixes some non-zero element of $X$ (By Orbit-Stabiliser, since $|X| = p^2$ and $|\langle g\rangle|=p$). In an exercise, I am then asked to deduce from this the statement above, which I am stuck on.
 A: Here is a solution using linear algebra over $\mathbb F_p$.
$g$ satisfies $0=g^p-I=(g-I)^p$.
Therefore, $1$ is the only eigenvalue of $g$.
Since $g$ is a $2\times 2$ matrix, these are the only possible Jordan forms for $g$:
$$
\begin{bmatrix}1&0\\0&1\end{bmatrix}
\quad\text{and}\quad
\begin{bmatrix}1&1\\0&1\end{bmatrix}
$$
But $I$ does not have order $p$.
A: It sounds like you want a mixture of group actions and linear algebra.

Let $g\in G=GL_2(\Bbb{F}_p)$ be an element of order $p$. Let's denote $H=\langle g\rangle$, and $X=\Bbb{F}_p^2$ the set of (column) all vectors on which both $H$ and $G$ act by matrix multiplication from the left.


*

*By the orbit-stabilizer theorem the orbits of $H$ have sizes $1$ and $p$ only.

*Because the zero vector forms an $H$-orbit of size $1$ and $|X|=p^2$, there must be other $H$-orbits of size $1$. Let $x\in X$ form such a singleton orbit of $H$.

*The scalar multiples of $x$ are then all eigenvectors of $g$ belonging to eigenvalue $\lambda=1$. Let $y\in X$ be a vector that is linearly independent from $x$. Then $y$ cannot belong to the eigenvalue $1$ for then we would have $g=1_G$.

*The vectors $x$ and $y$ form a basis $\mathcal{B}$ of $X$ over $\Bbb{F}_p$, so we have $g\cdot x=x$ and $g\cdot y=ax+by$ for some $a,b\in\Bbb{F}_p$. The matrix of
$g$ with respect to $\mathcal{B}$ looks like
$$
M_{\mathcal{B}}(g)=\left(\begin{array}{cc}
1&a\\
0&b
\end{array}\right).
$$

*We have $\det(g)=b$, so $1=\det(g^p)=b^p=b$.

*Because $g\neq 1_G$ we have $a\neq0$.

*With respect to the basis $\mathcal{B}'=\{ax,y\}$ the matrix of $g$ thus looks like
$$
M_{\mathcal{B}'}(g)=\left(\begin{array}{cc}
1&1\\
0&1
\end{array}\right).
$$
A: This gives a solution by group action theory, as you wish, but I couldn't avoid using the result (*) hereunder, which is seemingly as much non-elementary as the tools in the other answers/comments (for its proof, see in this site e.g. here).

Let's define $X:=\{M\in \operatorname{GL}_2(\mathbb{F}_p)\mid M^p=I\}$
Lemma. $G\in \operatorname{GL}_2(\mathbb{F}_p)\wedge M\in X \Longrightarrow GMG^{-1}\in X$.
Proof. $(GMG^{-1})^p=GM^pG^{-1}=GIG^{-1}=GG^{-1}=I \Longrightarrow GMG^{-1}\in X$.
$\Box$
Therefore, $\operatorname{GL}_2(\mathbb{F}_p)$ acts by conjugation on $X$. Note that $\tilde M:=\begin{bmatrix}1&1\\0&1\end{bmatrix}\in X$, because $\tilde M^p=I$, and thence we can apply the Orbit-Stabilizer Theorem to $\tilde M$:
$$|O(\tilde M)||\operatorname{Stab}(\tilde M)|=|\operatorname{GL}_2(\mathbb{F}_p)|=(p^2-1)(p^2-p)=p(p+1)(p-1)^2 \tag 1$$
Now, $\operatorname{Stab}(\tilde M)=\{G\in \operatorname{GL}_2(\mathbb{F}_p)\mid G\tilde M=\tilde MG\} = \Biggl\{G\in \operatorname{GL}_2(\mathbb{F}_p)\mid G=\begin{bmatrix}a&b\\0&a\end{bmatrix}, a,b\in \mathbb{F}_p\Biggr\}$, whence $|\operatorname{Stab}(\tilde M)|=p(p-1)$ and finally, by $(1)$, $|O(\tilde M)|=(p+1)(p-1)=p^2-1$. Now, $|X|=p^2-1$ (*) and $O(\tilde M) \subseteq X$, whence $O(\tilde M)=X$, and the action is transitive.
A: You have shown that all elements of order $p$ are in the same conjugacy class.  But $\begin{bmatrix}1&1\\0&1\end{bmatrix}$ has order $p$.
