For starters consider the following candidate factorization:
This works if $a+b\equiv 12$ and $ab\equiv 16$. The discriminant of the quadratic
is equal to $20$. So if $20$ is a quadratic residue modulo $p$, then we have a factorization
of this kind. As $20=2^2\cdot5$ we are really interested in, whether $5$ is a quadratic residue modulo $p$ or not. By quadratic reciprocity this happens, iff $p$ is a quadratic residue modulo $5$. So we get that a factorization of this kind exists, iff $p\equiv\pm1\pmod5.$ Of course, this is equivalent to $p$ being congruent to either $1,9,11$ or $19$ modulo $20$.
I don't know, if it is easy or difficult to extend this elementary argument and detect, whether these quadratic factors will split further. Also, this argument obviously doesn't rule out the possibility of other quadratic factors, but you seemed to have the cases $p\equiv\pm2\pmod 5$ covered already.