What is the coefficient of the term $x^3y^5$, as a result of the binomial expansion of the following term? We have the term
$(1+xy+y^2)^n$ 
If we expand it using the binomial theorem, why is the factor of the term $x^3y^5$ the following: $4{n\choose 4}$? (The binomial coefficient n choose 4 multiplied by 4)
 A: $\newcommand{\bbx}[1]{\,\bbox[15px,border:1px groove navy]{\displaystyle{#1}}\,}
 \newcommand{\braces}[1]{\left\lbrace\,{#1}\,\right\rbrace}
 \newcommand{\bracks}[1]{\left\lbrack\,{#1}\,\right\rbrack}
 \newcommand{\dd}{\mathrm{d}}
 \newcommand{\ds}[1]{\displaystyle{#1}}
 \newcommand{\expo}[1]{\,\mathrm{e}^{#1}\,}
 \newcommand{\ic}{\mathrm{i}}
 \newcommand{\mc}[1]{\mathcal{#1}}
 \newcommand{\mrm}[1]{\mathrm{#1}}
 \newcommand{\pars}[1]{\left(\,{#1}\,\right)}
 \newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}}
 \newcommand{\root}[2][]{\,\sqrt[#1]{\,{#2}\,}\,}
 \newcommand{\totald}[3][]{\frac{\mathrm{d}^{#1} #2}{\mathrm{d} #3^{#1}}}
 \newcommand{\verts}[1]{\left\vert\,{#1}\,\right\vert}$
\begin{align}
&\bbox[#ffd,10px]{\bracks{x^{3}}\bracks{y^{5}}\pars{1 + xy + y^{2}}^{n}}  \\[5mm] = &\
\bracks{x^{3}}\bracks{y^{5}}\sum_{a,b,c\ \in\ \mathbb{N}_{\geq\ 0}}{n! \over a!\, b!\, c!}\,
1^{a}\pars{xy}^{b}\pars{y^{5}}^{c}\bracks{a + b + c = n}
\\[5mm] = &\
\bracks{x^{3}}\bracks{y^{5}}\sum_{a,b,c\ \in\ \mathbb{N}_{\geq\ 0}}{n! \over a!\, b!\, c!}\,
x^{b}y^{b + 2c}\bracks{a + b + c = n}
\\[5mm] = &\
\sum_{a,b,c\ \in\ \mathbb{N}_{\geq\ 0}}{n! \over a!\, b!\, c!}\,
\bracks{b = 3}\
\underbrace{\bracks{b + 2c = 5}\bracks{a + b + c = n}}
_{\ds{\implies \pars{~c = 1\ \mbox{and}\ a = n - 4}~}}
\\[5mm] = &\
{n! \over \pars{n - 4}!\, 3!\, 1!} = {n! \over \pars{n - 4}!\, 4!}\,{4! \over 3!} = \bbx{4{n \choose 4}}
\end{align}
A: Yes, for n = 5, use the binomial theorem by substituting $xy + y^2$ for "$a$" and then expand $(1 + a)^5 = 1 + 5a + 10a^2 + 10a^3 + 5a^4 + a^5$.
Then substitute back $xy + y^2$ for $a$.........
Here, the only term which will give $x^3y^5$ is $5a^4$ whereby again using the binomial theorem on $5(xy + y^2)^4$ we get......
$5x^4y^4 + 5(4)x^3y^5 + 5(6)x^2y^6 + 5(4)xy^7 + 5y^8$
We get $20x^3y^5$ which is in keeping with $4\binom{5}{4}$
So to answer you question in a general sense, expanding a trinomial involves more than one application of the binomial theorem for some of the terms. 
A: It is convenient to use the coefficient of operator $[x^n]$ to denote the coefficient of $x^n$ in a series.

We obtain
  \begin{align*}
\color{blue}{[x^3y^5]}&\color{blue}{(1+xy+y^2)}\\
&=[x^3y^5]\sum_{j=0}^n\binom{n}{j}(xy)^j(1+y^2)^{n-j}\tag{1}\\
&=[y^5]\binom{n}{3}y^3(1+y^2)^{n-3}\tag{2}\\
&=\binom{n}{3}[y^2]\sum_{j=0}^{n-3}\binom{n-3}{j}y^{2j}\tag{3}\\
&=\binom{n}{3}\binom{n-3}{1}\tag{4}\\
&=\frac{n(n-1)(n-2)}{3!}\cdot(n-3)\\
&=4\cdot\frac{n(n-1)(n-2)(n-3)}{4!}\\
&\,\,\color{blue}{=4\binom{n}{4}}
\end{align*}

Comment:


*

*In (1) we apply the binomial theorem to $(xy+(1+y^2))^n$. This way we can easily select the coefficient of $x^3$.

*In (2) we select the coefficient of $x^3$.

*In (3) we use the formula $[y^{p-q}]A(y)=[y^p]y^qA(y)$ and apply the binomial theorem again.

*In (4) we select the coefficient of $y^2$.
