How does one treat big O when Taylor expanding $\sin(\sin x)$? I am working on finding the Taylor (Maclaurin expansion) of $\sin(\sin(x))$ to the third order.
We have $$\sin{t} = t - \frac{t^3}{6} + \mathcal{O}(t^5)$$
If I then set $t = \sin(x)$, and expand once more, I end up with a big O which is
$$\mathcal{O}((x - \frac{x^3}{6} + \mathcal{O}(x^5))^5)$$
How does one treat this mathematically? How do I know which order the error will be in my final expansion?
 A: The first line can be written as $\sin t = t - \frac{t^3}{6} + h(t) t^5$ for some function $h$ such that $|h(t)| \le C$ for $t$ near $0$.
Then,
\begin{align}
\sin \sin x
&= \sin x  - \frac{(\sin x)^3}{6} + h(\sin x) (\sin x)^5
\\
&= \left(x - \frac{x^3}{6} + h(x) x^5\right)
- \frac{1}{6} \left(x - \frac{x^3}{6} + h(x) x^5\right)^3
+ h\left(x - \frac{x^3}{6} + h(x) x^5\right)
\left(x - \frac{x^3}{6} + h(x) x^5\right)^5
\end{align}

*

*The first term contributes $x - x^3/6$. The $h(x) x^5$ is in $O(x^5)$ by definition, since $h(x)$ is bounded for $x$ near $0$.

*Upon expanding the cube, the second term contributes $-\frac{1}{6} x^3$. All other terms have at least a $x^5$ term, and together are still in $O(x^5)$.

*The factor $h\left(x - \frac{x^3}{6} + h(x) x^5\right)$ is bounded for $x$ sufficiently close to zero since the expression $x - \frac{x^3}{6} + h(x) x^5$ tends to zero as $x \to 0$. Upon expanding $\left(x - \frac{x^3}{6} + h(x) x^5\right)^5$, all terms have at least a $x^5$ term, so the whole expression is $O(x^5)$.

A: Expanding only the terms to the third order, $$\sin(\sin x)\\
 =  x - \frac{x^3}{6} + \mathcal{O}(x^5) - \frac{x^3\left(1 - \dfrac{x^2}{6} + \mathcal{O}(x^4)\right)^3}{6} + \mathcal{O}\left(x^5\left(1- \frac{x^2}{6} + \mathcal{O}(x^4)\right)^5\right)\\
=x-\frac{x^3}{6}-\frac{x^3}6+\mathcal O(x^5).$$

Check:
Using the chain rule,
$$\frac{df}{dx}=\cos t\cos x\to1$$
$$\frac{d^2f}{dx^2}=-\sin t\cos^2x-\cos t\sin x\to0$$
$$\frac{d^3f}{dx^3}=-\cos t\cos^3x+2\sin t\cos x\sin x+\sin t\sin^2 x-\cos t\cos x\to-2.$$
