Rudin 5.11. Second order derivative 
I got a slightly different result than suggested
$$\begin{align}
f''(x)&=\lim_{h\rightarrow 0}\frac{f'(x+h)-f'(x-h)}{2h} &(1)\\
&=\lim_{h\rightarrow 0}\frac{\frac{f(x+h)-f(x)}{h}-\frac{f(x)-f(x-h)}{h}}{2h}&(2)\\
&=\lim_{h\rightarrow 0}\frac{f(x+h)+f(x-h)-2f(x)}{2h^2}&(3)
\end{align}$$
It seems that I got an extra factor of 2. I realized that such proof is not rigorous, because from (1) to (2), there are double limits and something mysterious could happen. Is it the reason why I got a wrong result?
 A: As you mentioned, the step from (1) to (2) is a little suspicious, especially since by definition,
\begin{equation}
f'(x+h) = \lim_{k \to 0} \dfrac{f(x+h+k) - f(x+h)}{k}.
\end{equation}
The most common, and the most easily generalizable approach to this question would be to use a Taylor expansion:
\begin{align}
f(x+h) = f(x) + h f'(x) + \dfrac{h^2}{2} f''(x) + \varphi(h), \tag{$*$}
\end{align}
for some function $\varphi$ (which is just LHS minus RHS) defined in a neighbourhood of $x$, such that $\lim_{h \to 0} \dfrac{\varphi(h)}{h^2} = 0$. The proof of the fact that $\varphi$ satisfies this limit condition can be found in Michael Spivak's Calculus, Chapter $20$, Theorem $1$ (or you can come up with your own proof; it's not that hard to prove... in the general case, just apply L'Hopital's rule several times, or you can give a proof by induction).
Similarly, we have
\begin{align}
f(x-h) = f(x) - h f'(x) + \dfrac{h^2}{2} f''(x) + \varphi(-h). \tag{$*$}
\end{align}
Can you see how adding, rearranging terms of $(*)$ and $(**)$ yields the desired result?
A: You made a slight mistake. Here, the corrected version.
$$\begin{align}
f''(x)&=\lim_{h\rightarrow 0}\frac{f'(x+h)-f'(x-h)}{2h} &(1)\\
&=\lim_{h\rightarrow 0}\frac{\frac{f(x+2h)-f(x)}{2h}-\frac{f(x)-f(x-2h)}{2h}}{2h}&(2)\\
&=\lim_{h\rightarrow 0}\frac{f(x+2h)+f(x-2h)-2f(x)}{4h^2}&(3)\\
&=\lim_{h'\rightarrow 0}\frac{f(x+h')+f(x-h')-2f(x)}{(h')^2}&(4)\\
\end{align}$$
where $h'=\frac{h}{2}$. Since, $h$ is small enough and it approaches $0$. Therefore, we can replace $h'$ by $h$ in $(4)$, and we get the desired formula.
