Yes, that's essentially the idea.
But this follows from the definition of continuity: we just need to show that if $f$ is continuous at $g(a)$ and $g$ is continuous at $a$, then $f\circ g$ is continuous at $a$. Because then both sides evaluate to $f(g(a))$, by definition of continuity (which requires that the limit exist and be equal to evaluating the function at the point.
To prove that if $g$ is continuous at $a$ and $f$ is continuous at $g(a)$, then $f\circ g$ is continuous at $a$, note first that $f\circ g$ is defined at $a$. Now, let $\epsilon\gt0$. then we know that there exists $\delta_1\gt 0$ such that if $|y-g(a)|\lt\delta_1$, then $|f(y)-f(g(a))|\lt\epsilon$; this holds, because $f$ is continuous at $g(a)$.
Now, since $g$ is continuous at $a$, and $\delta_1\gt 0$, this means that there exists $\delta\gt 0$ such that if $|x-a|\lt\delta$, then $|g(x)-g(a)|\lt\delta_1$.
Thus, if $|x-a|\lt\delta$, then $|g(x)-g(a)|\lt\delta_1$, and therefore $|f(g(x))-f(g(a))|=|f\circ g(x) - f\circ g(a)|\lt\epsilon$.
Thus, for all $\epsilon\gt 0$ there exists $\delta\gt 0$ such that if $|x-a|\lt\delta$, then $|f\circ g(x) - f\circ g(a)|\lt\epsilon$. Therefore, $f\circ g$ is continuous at $a$.
Therefore, we have that $\lim\limits_{y\to g(a)} f(y) = f(g(a))$, since $f$ is continuous at $g(a)$; and $\lim\limits_{x\to a}f\circ g(x) = f\circ g(a) = f(g(a))$, since $f\circ g$ is continuous at $a$.
You don't quite need $g$ to be continuous: if $\lim\limits_{x\to a}g(x)=L$ and $f$ is continuous at $L$, then we have
$$\lim_{x\to a}f\circ g(x) = \lim_{y\to L}f(y) = f(L).$$
To verify this, let $\epsilon\gt 0$. Then there exists $\delta_1\gt 0$ such that for all $x$, $|y-L|\lt\delta_1$ implies $|f(y)-f(L)|\lt\epsilon$. Since $\lim\limits_{x\to a}g(x)=L$, there exists $\delta\gt 0$ such that if $0\lt |x-a|\lt \delta$ then $|g(x)-L|\lt\delta_1$. So, suppose that $0\lt |x-a|\lt\delta$. Then $|g(x)-L|\lt\delta_1$, and therefore $|f(g(x))-f(L)|\lt\epsilon$. Therefore, for every $\epsilon\gt 0$ there exists $\delta\gt 0$ such that if $0\lt |x-a|\lt\delta$, then $|f(x)-f(L)|\lt\epsilon$. This proves that if $\lim\limits_{x\to a}g(x) = L$ and $f$ is continuous at $L$, then
$$\lim\limits_{x\to a}f\circ g(x) = \lim\limits_{y\to L}f(y) = f(L).$$
In particular, if $g$ is continuous at $a$, then we replace $L$ with $g(a)$.
We cannot omit the continuity of $f$ at $L$, though: take $g(x) = 0$ for all $x$, and let
$$f(x) = \left\{\begin{array}{ll}
1 &\text{if }x\neq 0\\
0 &\text{if }x=0.
\end{array}\right.$$
Then $\lim\limits_{x\to a}f(g(x)) = 0$, because $f(g(x))=f(0)=0$. But
$$\lim\limits_{y\to 0}f(y) = 1,$$
because we never take the value $y=0$ in evaluating the limit.
One can replace continuity of $f$ with other conditions; for example, we may ask that $g$ have a limit $L$ at $a$, and moreover, that for every $\delta\gt0$ there exist an $\eta\gt 0$ such that $g$ takes all values on $(L-\eta,L+\eta)$, except perhaps $L$ itself, on $(a-\delta,a+\delta)-\{a\}$.
Added. The situation with limits as $x\to\infty$ is essentially the same if $\lim\limits_{x\to\infty}g(x)$ exists and is real. It is more complicated when the limit of $g$ is $\pm\infty$. See this answer for some discussion on that.