Notice that the calculation which you did actually shows that $$\frac{\binom{p^2}{p}}{p}=\frac{p^2\cdot x}{p^2(p-1)!}=\frac{x}{(p-1)!}\equiv 1\pmod{p^2}.$$ That is, $\frac{\binom{p^2}{p}}{p}-1$ is divisible by $p^2$. Thus $\binom{p^2}{p}-p$ is divisible by $p^3$, so $\binom{p^2}{p}\equiv p\pmod{p^3}$.
In fact, by a more refined calculation, you can show that $\binom{p^2}{p}\equiv p\pmod{p^4}$ as well. To make it easier to understand what's going on, let's write $y=p^2$ and expand $$x=(y-1)(y-2)\cdots(y-(p-1)).$$ Grouping terms according to their power of $y$, we get $$x=(-1)^{p-1}(p-1)!+(-1)^{p-2}\left(\frac{(p-1)!}{1}+\frac{(p-1)!}{2}+\dots+\frac{(p-1)!}{p-1}\right)y+\text{terms with $y^2$ and higher powers of $y$}.$$ Let's now think about the integer $$s=\frac{(p-1)!}{1}+\frac{(p-1)!}{2}+\dots+\frac{(p-1)!}{p-1}$$ mod $p$. Mod $p$, we can think of the $k$th term of $s$ as $(p-1)!$ multiplied by the multiplicative inverse of $k$ mod $p$. The multiplicative inverses of $1,\dots,p-1$ mod $p$ are just all the nonzero residues mod $p$ in some different order, so $$s\equiv (p-1)!(1+2+\dots+(p-1))\pmod{p}.$$ Since $p$ is odd, $1+2+\dots+(p-1)=\frac{p(p-1)}{2}$ is divisible by $p$, so $s$ is divisible by $p$.
Now consider $x$ mod $p^3$. Since $y=p^2$, all the terms of $x$ with $y^2$ or higher powers are $0$ mod $p^3$. Since $s$ is divisible by $p$, the term $(-1)^{p-2}sy$ is also $0$ mod $p^3$. Thus $$x\equiv (-1)^{p-1}(p-1)!=(p-1)!\pmod{p^3}$$ and now as in the argument above we find that $$\binom{p^2}{p}\equiv p\pmod{p^4}.$$
