coefficient of $x^{17}$ in $(x^2+x^3+x^4+x^5+x^6+x^7)^3$ coefficient of $x^{17}$ in $(x^2+x^3+x^4+x^5+x^6+x^7)^3$ - I'm reviewing for an exam and don't understand the answer, C(11+3−1,11)−C(3,1)×C(5+3−1,5).
 A: 
Using the coefficient of operator $[x^n]$ to denote the coefficient of $x^n$ we obtain
\begin{align*}
\color{blue}{[x^{17}](x^2+x^3+\cdots+x^7)^3}&=[x^{17}]x^6(1+x+\cdots+x^5)^3\tag{1}\\
&=[x^{11}]\left(\frac{1-x^6}{1-x}\right)^3\tag{2}\\
&=[x^{11}]\left(1-\binom{3}{1}x^6\right)\sum_{n=0}^\infty \binom{n+3-1}{n}x^n\tag{3}\\
&=\left([x^{11}]-\binom{3}{1}[x^{5}]\right)\sum_{n=0}^\infty \binom{n+3-1}{n}x^n\\
&\color{blue}{=\binom{11+3-1}{11}-\binom{3}{1}\binom{5+3-1}{5}}\tag{4}\\
&=\binom{13}{2}-3\binom{7}{2}\\
&=15
\end{align*}

Comment:


*

*In (1) we factor out $x^6$.

*In (2) we apply the rule $[x^p]x^qA(x)=[x^{p-q}]A(x)$ and use the finite geometric series formula.

*In (3) we expand $(1-x^6)^3$ up to $x^6$ since other terms do no contribute to $x^{11}$.

*In (4) we select the coefficient of $x^{11}$ and $x^5$.
A: $$
\begin{align}
\left[x^{17}\right]\left(x^2+x^3+x^4+x^5+x^6+x^7\right)^3
&=\left[x^{11}\right]\left(1+x+x^2+x^3+x^4+x^5\right)^3\\
&=\left[x^{11}\right]\left(\frac{1-x^6}{1-x}\right)^3\\
&=\left[x^{11}\right]\sum_{j=0}^3(-1)^j\binom{3}{j}x^{6j}
\sum_{k=0}^\infty(-1)^k\binom{-3}{k}x^k\\
&=\left[x^{11}\right]\sum_{j=0}^3(-1)^j\binom{3}{j}x^{6j}
\sum_{k=0}^\infty\binom{k+2}{k}x^k\\
&=\sum_{j=0}^1(-1)^j\binom{3}{j}\binom{13-6j}{11-6j}\\
&=\binom{3}{0}\binom{13}{11}-\binom{3}{1}\binom{7}{5}\\[9pt]
&=15
\end{align}
$$
