In the book I'm using, $A(x)$ denotes the formal power series (generating function), $A(x) = \sum a_ix^i$. I'm really stuck on this problem. Thanks for any help.
My attempt after the given hint: $$\begin{align} A(x)&=\sum_{n\geq 0} \binom{n}{2}x^n\\ &=\sum_{n\geq 0} \frac{x^2}{2}\frac{d^2}{dx^2}(x^{n})\\ &=\frac{x^2}{2} \frac{d^2}{dx^2}\sum_{n\geq 0} x^{n}\\ &=\frac{x^2}{2} \frac{d^2}{dx^2}\frac{1}{1-x}\\ &=\frac{x^2}{2} \frac{2}{(1-x)^3} \end{align}$$ Sorry, I'm new to $\rm \LaTeX$.