The facts that $\mathbb F_{q'}$ is a field extension of $ \mathbb F_{q}$ if and only if $q'$ is a power of $q$, that there is a 'unique' extension with given cardinality or degree, and therefore that all extensions are normal (irreducible poly with a root in the extension factors completely) make this problem, as asked, pretty straight-forward almost without calculation, though I get less than you did with calculation.
By calculation (!), $P$ has no roots over ${\mathbb F}_3$. Hence it cannot have an irreducible cubic factor, which means that the polynomial cannot
factor over $\mathbb F_{27}$, as otherwise it would have to do completely.
Hence $P$ must factor completely over $\mathbb F_{81}$ because either a root of $P$ generates the 'unique' extension of degree 4 over $\mathbb F_3$, or $P$ factored over $\mathbb F_3$ as quadratic polynomials, and so factored completely in $\mathbb F_9$ (which you ruled you by calculation - but...)
Either way $P$ has to factor in $\mathbb F_9$, because we know that it factors completely in $\mathbb F_{81}$, which is the 'unique' extension of degree 2 over $\mathbb F_9$.