Find the coefficient of $x^n$ in generating function $\frac{1}{(1-x)^3(1+x)}$ One of the problems in my Discrete Math course states that we need to find the coefficient of $x^k$ in generating function $f(x)=\frac{1}{(1-x)^3(1+x)}$. The answer is $$\frac{1}{8}+\frac{1}{4}(k+1)+\frac{1}{2}\binom{k+2}{2}+\frac{(-1)^k}{8}$$ The solution was using partial fraction decomposition. I wanted to use the identities for the formal power series. For example: 
$$\frac{1}{1+x}=\frac{1}{1-(-x)}=\sum_{k=0}^\infty \binom{n-1+k}{k}(-1)^kx^k$$
If $x^k=x^0$ then:
$$\binom{1-1+0}{0}x^0=\binom{0}{0}x^0=1$$
If $x^k=x^1$ then:
$$-\binom{1-1+1}{1}x^1=-\binom{1}{1}x^1=-1$$
Therefore since:
$$\frac{1}{(1-x)^3}=\sum_{k=0}^\infty \binom{n-1+k}{k}x^k$$
Hence the coefficient of $x^k$ would be:
$$\binom{0}{0}x^0\binom{3-1+k}{k}x^k-\binom{1}{1}x^1\binom{3-1+k-1}{k-1}x^{k-1}=$$
$$=\binom{k+2}{k}-\binom{k+1}{k-1}$$
When check my solution it doesn't equal to the textbook's solution. I really want to use identities for such problems, can I still use them and if yes what is the correct identity or what am I doing wrong?
 A: We have
$$\frac1{1+x}=\sum_{n\ge 0}(-1)^nx^n$$
and
$$\frac1{(1-x)^3}=\sum_{n\ge 0}\binom{n+2}2x^n\;,$$
so
$$\frac1{(1+x)(1-x)^3}=\left(\sum_{n\ge 0}(-1)^nx^n\right)\left(\sum_{n\ge 0}\binom{n+2}2x^n\right)\;.\tag{1}$$
It’s at this point that you went astray. The coefficent of $x^n$ in $(1)$ is
$$\begin{align*}
\sum_{k=0}^n(-1)^k\binom{(n-k)+2}2&=\sum_{k=0}^n(-1)^k\binom{n+2-k}2\\
&=\sum_{\ell=2}^{n+2}(-1)^{n+2-\ell}\binom{\ell}2&&\text{setting }\ell=n+2-k\\
&=(-1)^n\sum_{\ell=2}^{n+2}(-1)^\ell\binom{\ell}2\;,
\end{align*}$$
which is probably not something for which you know a closed form. It turns out that
$$\begin{align*}
(-1)^n\sum_{\ell=2}^{n+2}(-1)^\ell\binom{\ell}2&=(-1)^n(-1)^n\left\lfloor\frac{(n+2)^2}4\right\rfloor\\
&=\left\lfloor\frac{(n+2)^2}4\right\rfloor\;.
\end{align*}$$
It is not obvious that this is equal to
$$\frac18+\frac14(n+1)+\frac12\binom{n+2}2+\frac{(-1)^n}8\;,$$
but in fact
$$\begin{align*}
\frac{1+(-1)^n}8+\frac{n+1}4+\frac12\binom{n+2}2&=\frac{1+(-1)^n}8+\frac{(n+1)(n+3)}4\\
&=\frac{1+(-1)^n}8+\frac{(n+2)^2-1}4\\
&=\frac{(n+2)^2}4+\frac{(-1)^n-1}8\\
&=\begin{cases}
\frac{(n+2)^2}4,&\text{if }n\text{ is even}\\
\frac{(n+2)^2-1}4,&\text{if }n\text{ is odd}
\end{cases}\\
&=\left\lfloor\frac{(n+2)^2}4\right\rfloor\;.
\end{align*}$$
