find the coefficient of $x^5$ in $(1+2x-3x^2)^6$ I know that I should first factor the expression within the brackets to $-(3x+1)^6(x-1)^6$. However, after that I do not know what to do.
 A: Actually you can do it without factorization as well. Just like binomial theorem, we have a multinomial theorem as well. For example,
$$(a+b+c)^n=\sum_{\substack{i,j,k\\i+j+k=n}}\frac{n!}{i!j!k!}a^ib^jc^k.$$
This can be generalized to higher order multinomials as well.
So in your case,
$$(1+2x-3x^2)^6=\sum_{\substack{i,j,k\\i+j+k=6}}\frac{6!}{i!j!k!}(1)^i\,(2x)^j\,(-3x^2)^k=\sum_{\substack{i,j,k\\i+j+k=6}}\frac{6!}{i!j!k!}2^j(-3)^k\,x^{\color{red}{j+2k}}.$$
For the coefficient of $x^5$, we want $j+2k=5$ and $i+j+k=6$. This means $i+(5-k)=6 \implies i-k=1.$
Thus
$$(i,j,k) \in \{(1,5,0), (2,3,1), (3,1,2)\}.$$
Thus the coefficient of $x^5$ is
$$\frac{6!}{1!5!0!}2^5(-3)^0+\frac{6!}{2!3!1!}2^3(-3)^1+\frac{6!}{3!1!2!}2^1(-3)^2=\color{blue}{-168}.$$
A: There shouldn't be a minus sign outside, it's wiped out when you take the sixth power.
$(1+2x-3x^2)^6 = (3x+1)^6(x-1)^6$
Now just find all the ways you can get to a power $5$ in the result. $5 = 0+5 = 1+4 = 2+3 = 3+2 = 4+1 = 5+0$.
So you need to find only the relevant terms in each of the individual expansions. As an example here is "$3+2$":
$\binom 6 3 (3x)^3(1)^3 \cdot \binom 6 2(x)^2(-1)^4$.
Compute all similar terms and add them up.
A: We can also apply the binomial theorem twice in order to find the coeffcient. It is convenient to use the coefficient of operator $[x^j]$ to denote the coefficient of $x^j$ of an expression.

We obtain
\begin{align*}
\color{blue}{[x^5]}&\color{blue}{(1+2x-3x^2)^{6}}\\
&=[x^5]\sum_{j=0}^{6}\binom{6}{j}(-3x^2)^j(1+2x)^{6-j}\tag{1}\\
&=\sum_{j=0}^2\binom{6}{j}(-3)^j[x^{5-2j}]\sum_{k=0}^{6-j}\binom{6-j}{k}(2x)^k\tag{2}\\
&=\sum_{j=0}^2\binom{6}{j}(-3)^j\binom{6-j}{5-2j}2^{5-2j}\tag{3}\\
&=\binom{6}{0}(-3)^0\binom{6}{5}2^5+\binom{6}{1}(-3)^1\binom{5}{3}2^3+\binom{6}{2}(-3)^2\binom{4}{1}2^1\\
&=192-1\,440+1\,080\\
&\,\,\color{blue}{=-168}
\end{align*}

Comment:

*

*In (1) we apply the binomial theorem once to $((1+2x)-3x^2))^6$.


*In (2) we apply the rule $[x^p]x^qA(x)=[x^{p-q}]A(x)$ and we set the upper index of the outer sum to $2$, since other terms do not contribute. We also apply the binomial theorem once again.


*In (3) we select the coefficient of $x^{5-2j}$.
