In order to factor $a^4-3a^2-2ab+1-b^2$, I find that $a=1, b=-1$ makes the value of the expression 0. Thus, I assume $b=-a$.
I rewrite the expression on the assumption as: $$a^4-3a^2-2ab+1-b^2$$ $$=a^4-3a^2+2a^2+1-a^2$$ $$=a^4-2a^2+1$$ $$=(a^2-1)^2$$
Then I insert $a+b$, which is another form of the assumption above, into one of the factors. Since $a+b=0$, this insertion should cause no change to whatever relationships in the original expression. I arbitrarily set its coefficient as $1$. A factor, therefore, would be $(a^2+a+b-1)$.
I then divide the original expression by the factor and find that it indeed can be divided without a remainder and I also find the other factor. Thus, $$a^4-3a^2-2ab+1-b^2$$ $$=(a^2+a+b-1)(a^2-a-b-1)$$ Q.E.D.
The problem is that I don't know what I am doing. I tried to use the factor theorem for a bivariate (if this is the proper word for it) expression but I might be fabricating a fortuitous reasoning from the answer. Yes, I knew the answer before I tried, and no, I don't know how I can solve this in other ways. It would be great if someone can tell me if my reasoning can be made more clear.