Can you verify my ideas here with the chain rule using limits? I've been working on understanding limits thoroughly, so I'm rewriting how I understand the chain rule.  Please help me fill in my gaps in understanding.
$f$ is some function. Then
$f'(x) = \lim\limits_{h \to 0}\frac{f(x+h)-f(x)}{h}$
Now I might want to evaluate something like 
$$\left(f(g(x))\right)' = \lim\limits_{h \to 0}\frac{f(g(x+h))-f(g(x))}{h}$$
Evaluating this is tricky, so we need a way to do it.  
$\left(f(g(x))\right)' = (\lim\limits_{h \to 0}\frac{f(g(x+h))-f(g(x))}{g(x+h)-g(x)}\cdot\frac{g(x+h)-g(x)}{h}$)
$\left(f(g(x))\right)' = (\lim\limits_{h \to 0}\frac{f(g(x+h))-f(g(x))}{g(x+h)-g(x)})\cdot\lim\limits_{h \to 0}(\frac{g(x+h)-g(x)}{h})$
if $k=g(x+h)-g(x)$, then
$g(x+h)=k+g(x)$, so
$\left(f(g(x))\right)' = (\lim\limits_{h \to 0}\frac{f(g(x)+k)-f(g(x))}{k})\cdot(\lim\limits_{h \to 0}\frac{g(x+h)-g(x)}{h})$
and, assuming I can go ahead and just change the limit variable on the left term, then
$=(\lim\limits_{k \to 0}\frac{f(g(x)+k)-f(g(x))}{k}) \cdot (\lim\limits_{h \to 0}\frac{g(x+h)-g(x)}{h})$
$=f'(g(x))\cdot g'(x)$
Which is easier to figure out, and is also the chain rule.
Is that correct?
 A: The final idea is generally correct, as Michael Hardy points out, modulo some refinements to deal with tricky situations.
However, I wanted to point out that some of the arguments leading up to that final part are incorrect:


*

*You assume that
$$b\lim_{h\to 0}\frac{a}{h} = \lim_{h\to 0}\frac{ab}{h}.$$
This will work if $b$ does not depend on $h$ (in the sense that either both sides exist and are equal, or both sides do not exist), but cannot hold if $b$ does depend on $h$. To see that it cannot hold when $b$ depends on $h$, note that the right hand side will not depend on $h$ (because it's the value of a limit over $h$), but the left hand side does depend on $h$ (since $b$ depends on $h$ and is outside the limit, so the left hand side will be a function of $h$, namely $b$, times a constant, namely the value of the limit).

*For the same reason, when you write:
$$
\left(\lim\limits_{h \to 0}\frac{f(g(x)+k)-f(g(x))}{h}\right) \cdot \frac{k}{k}
=\left(\lim\limits_{h \to 0}\frac{f(g(x)+k)-f(g(x))}{k}\right) \cdot \frac{k}{h}$$
you run into trouble, because you cannot just pull the $h$ out of the limit that is a limit over $h$, and because $k$ depends on $h$, so you cannot move it in and out of a limit over $h$ as you would a constant.
What you need in both cases is the "product rule for limits": if
$$\lim_{h\to 0}\;k\quad\text{and}\quad \lim_{h\to 0}\;m\quad\text{both exist, then }\lim_{h\to 0}\;km = \left(\lim_{h\to 0}\;k\right)\left(\lim_{h\to 0}\;m\right),$$
which is what you use in the final paragraphs. 

We want to find
$$\lim_{h\to 0}\frac{f(g(x+h)) - f(g(x))}{h}.$$
You propose defining $k(h)$ (and it's important to keep the dependence on $h$ explicit, to prevent the errors you fell into) by
$$k(h) = g(x+h)-g(x)$$
so that we can rewrite
$$\frac{f(g(x+h)) - f(g(x))}{h} = \frac{f(g(x)+k(h)) - f(g(x))}{h}.$$
Nothing wrong here. Intuitively, what we want to then do is rewrite again into
$$\begin{align*}
\frac{f(g(x)+k(h)) - f(g(x))}{h} &= \frac{f(g(x)+k(h)) -f(g(x))}{k(h)}\cdot\frac{k(h)}{h}\\ 
&= \frac{f(g(x)+k(h)) - f(g(x))}{k(h)}\cdot \frac{g(x+h)-g(x)}{h},
\end{align*}$$
and then use the fact that a limit of a product is the product of the limits (if they both exist), and a change of variable in the first limit from $h$ to $k$, to deduce the Chain Rule.
The problem with this is that the rewriting is only valid if $k(h)\neq 0$ when $h\neq 0$; otherwise, the two functions are not equal, since they don't have the same domain. That is, we know that $k(h)=g(x+h)-g(x)$ is $0$ when $h=0$, but it's possible for $k(h)$ to equal zero for other values of $h$; and at those values, $\frac{f(g(x)+k(h))-f(g(x))}{h}$ is defined, but $\frac{f(g(x)+k(h))-f(g(x))}{k(h)}\cdot\frac{g(x+h)-g(x)}{h}$ is not, and we don't have equality.
If you think about it, though, at the places where $k(h)=0$, we have that
$$\frac{f(g(x)+k(h))-f(g(x))}{h}=0.$$
So the simple way to make this work is to use a different function instead of
$$\frac{f(g(x)+k(h)-f(g(x))}{k(h)}$$
on the right hand side, one that is equal to $f'(g(x))$ when $k(h)=0$ (the value we want to get), and is equal to the old value when $k(h)\neq 0$.
So we define a function $G(h)$ as follows:
$$G(h) = \left\{\begin{array}{ll}
\frac{f(g(x)+k(h))- f(g(x))}{k(h)} &\text{if }k(h)\neq 0\\
f'(g(x))&\text{if }k(h)=0.
\end{array}\right.$$
With this function, we do have that
$$\frac{f(g(x+h)) - f(g(x))}{h} = G(h)\cdot\frac{g(x+h)-g(x)}{h}$$
for all $h$. Since they take the same values at all values of $h$ (except $h=0$, where they are both undefined), the two have the same limit as $h\to 0$:
$$\lim_{h\to 0}\left(\frac{f(g(x+h))-f(g(x))}{h} \right)= \lim_{h\to 0}\left(G(h)\cdot\frac{g(x+h)-g(x)}{h}\right).$$
Now, 
$$\lim_{h\to0}\frac{g(x+h)-g(x)}{h} = g'(x),$$
so if $\lim\limits_{h\to 0}G(h)$ exists, then we'll be set.
Now, $k(h) = g(x+h)-g(x)$; as $h\to 0$, we have $k(h)\to 0$ because $g$ is continuous at $x$ (by virtue of being differentiable at $x$). 
If $k(h)=0$ for all values of $h$ near $0$, then 
$$\lim\limits_{h\to 0}G(h) = \lim_{h\to 0}f'(g(x)) = f'(g(x)).$$
If $k(h)\neq 0$ for all $h$ near $0$ (except at $h=0$), then we can do a change of variable: since $\lim\limits_{h\to 0}k(h) = 0$, we have:
$$\begin{align*}
\lim_{h\to 0}G(h) &= \lim_{h\to0}\frac{f(g(x)+k(h)) - f(g(x))}{k(h)}\\
&= \lim_{k(h)\to 0}\frac{f(g(x)+k(h)) - f(g(x))}{k(h)}\\
&= f'(g(x)).
\end{align*}$$
The justification for this formally requires epsilon and deltas, but the idea is: we can make $k(h)$ arbitrarily close to $0$ by making $h$ arbitrarily close to $0$, so taking the limit as $h$ approaches $0$ amounts to the same as taking the limit as $k(h)$ approaches zero.
What if $k(h)$ is not constant zero, but takes the value $0$ at arbitrarily small values of $h$ (that is, for every $\delta\gt 0$ we can find $h$ in $(-\delta,\delta)$ where $k(h)=0$, and we can find $h'$ in $(-\delta,\delta)$ where $k(h)\neq 0$)? Then one needs to argue carefully that the limit
$$\lim_{h\to 0}G(h)$$
is still equal to $f'(g(x))$; this is difficult to do informally, and there are no ready "limit rules" to help us. You would need to see the proof with epsilon and deltas. One can think of this as a rather "extreme and almost pathological" case, and the easy cases above follow the intuition you had fairly well, once you fix the problems with your manipulation of limits. 
Taking this for granted, we have:
$$\begin{align*}
\lim_{h\to 0}\frac{f(g(x+h))-f(g(x))}{h} &= \lim_{h\to 0}\frac{f(g(x)+k(h)) - f(g(x))}{h}\\
&= \lim_{h\to 0}\left(G(h)\cdot\frac{g(x+h)-g(x)}{h}\right)\\
&= \left(\lim_{h\to 0}G(h)\right)\left(\lim_{h\to 0}\frac{g(x+h)-g(x)}{h}\right)\\
&= f'(g(x))g'(x).
\end{align*}$$
A: This is correct except that it fails to address the fact that $k$ may be $0$ when $h$ is not.  If there is some neighborhood of $0$ such that when $h$ is in that neighborhood and $h\ne0$, then $k\ne0$, then everything above works fine.  If every open neighborhood of $0$ contains some value of $h\ne0$ for which $k=0$, then your argument fails to deal with that situation.  This is about the behavior of the function $g$, not about $f$.
The fraction
$$
\frac{f(g(x)+k)-f(g(x))}{k}
$$
is undefined when $k=0$, which may happen in some cases even when $h=0$.  Now consider the piecewise defined function
$$
k \mapsto \begin{cases} \frac{f(g(x)+k)-f(g(x))}{k} & \text{if }k\ne 0, \\  \\
f'(g(x)) & \text{if } k = 0.\end{cases} \tag{2}
$$
I think working with (2) in place of (1) can fill in the gap.  You still probably have to worry about the details of the logic, but that's the main idea.
