There are two natural definitions.
- It could mean continuous functions on $[a,b]$ which are smooth in the interior $(a,b)$ and so that all derivatives have limits at the end point $a$ and $b$.
- It could also mean functions on $[a,b]$ that can be extended to infinitely differentiable functions on a larger, open interval $(a-\varepsilon,b+\varepsilon)$.
Fortunately, the two are equivalent. It is rather obvious that a function which is smooth in the second sense is smooth in the first sense. The converse can be proved by appeal to the theorem that, for every sequence $(c_n)$ of real numbers, there is an infinitely differentiable function $g$ defined on a neighbourhood of $a$ so that $g^{(n)}(a)=c_n$ for all $n$. Given a function $f\in C^\infty([a,b])$ in the first sense, put $c_n=\lim_{x\to a}f^{(n)}(x)$, pick a $g$ by the theorem mentioned, and extend $f$ by using $g$ instead to the left of $a$. Do the same at $b$. The result is a smooth function on a bigger interval, so $f$ smooth on $[a,b]$ in the second sense.