You are correct. Second derivatives are not really the right thing to be considering. The main issue is that second (and other) derivatives are dependent on the parameterization of the curve, whereas "smoothness" is a geometric property that is independent of parameterization. We should really be considering parameterization-independent quantities like curvature and derivative of curvature with respect to arclength. But curvature is a nasty non-linear function, and using derivatives is much easier.
In fact, you might even say that the whole idea of cubic splines (enforcing continuity of second derivatives) is wrong. We ought to be enforcing continuity of curvature, instead. This has been done -- there are splines that have so-called "geometric continuity". This is an old idea, going back to Even Mehlum's KURGLA system in the 1970s. Again, it's more correct, but the computations are much more difficult, and the results often don't justify the effort.
The answers to this question have some further discussion.