I am trying to factor the polynomial $$(a-1)x^2 + a^2xy+(a+1)y^2.$$ The problem previous to it in the book uses the method of factoring a polynomial of the form $$ax^2 + bx +c$$ by inspection, and the problem following it uses a formula related to cubes (I thought it's best you know). That said, I began by multiplying the coefficients of $x^2$ and $y$, but that did not yield something good.So I started again by taking $ax$, $x$, and $y$ as common, and that yielded nothing good. I would show some of my other work, but that would seem way too messy without proper formatting.
Convince yourself that it's going to be $$(rx+sy)(tx+uy)$$ where $r$, $s$, $t$, and $u$ are going to have formulas involving $a$.
Note that $$rt=a-1,\quad su=a+1$$
How can you get two things that multiply to $a-1$? Don't look for anything really fancy; what are the simplest possibilities?
Same question for $a+1$.
Now you have some possibilities for $r$, $s$, $t$, and $u$; see which combination gives you the right coefficient for $xy$. One will soon note that r=a-1 and u=a+1 works fine.Putting the values gives us the result required.Hence,we get, (ax-x+y)(x+ay+y)
Look first at the coefficient of $xy$, $a^2$. The simplest way to get this from existing coefficients is $(a+1)(a-1)+1$. So one factor should contain $(a-1)x$ and the other should contain $(a+1)y$. Then the other terms need to have coefficient one to make the $xy$ coefficient correct in the product. That this all works out so nicely is deliberate, because generally such expressions will not factor nicely.