If $F$ is a completely arbitrary field (finite or not), then every finite-dimensional associative $F$-algebra $A$ without zero-divisors is a division algebra.
Indeed, given $a\neq0\in A$ the multiplication $\mu_a:A\to A:x\mapsto ax$ is an $F$- linear endomorphism.
It is injective because $a$ is not a zero-divisor (by the assumption on $A$) hence it is surjective by linear algebra.
Surjectivity implies that there exists $b\in A$ with $\mu_a(b)=1=ab $.
Similarly there exists $c\in A$ with $ca=1$.
But then $c(ab)=c1=c=(ca)b=1b=b$ so that $b=c$ is an inverse for $a$.
We have proved that $A$ is a division algebra since each nonzero element $a\in A$ is invertible.