All fields of prime order are isomorphic to the fields $\mathbb Z_p$. However, there are also other finite fields. Given $p$ prime and $n \in \mathbb N$, there is a field $\mathbb F_{p^n}$ of $p^n$ elements - this is quite advanced to prove, and some of these fields can be tricky to construct. We can also show that all finite fields have this form.
We can construct $\mathbb F_4$ as follows: take $\mathbb Z_2$ and add an element $\omega$ such that $\omega^2 + \omega + 1 = 0$ and $\omega^3 = 1$. We can show that the set of elements generated by $\mathbb Z_2 $ and $\omega $ form a field. However, it will not use modular arithmetic, as $\omega$ is not an integer.
$\mathbb F_4$ has elements $\{0,1,\omega, \omega^2\}$; note that $\omega + 1 = - \omega^2 = \omega^2$ (as $1 = -1$).
Recall that $\mathbb Z_p$ is a field if and only if $p$ is prime. There is no "modular arithmetic" field of order $4$.