Taylor's theorem on manifold Taylor's theorem for real-valued functions on manifolds is straightforward, and doesn't even require anything beyond differential structure. How does Taylor's theorem work for manifold-valued functions?
Suppose you have a function $f:\mathbb{R}\to M$, where $M$ is a manifold (i.e., $f$ is a curve on $M$). Is there some notion of a Taylor's theorem on $M$, i.e., a way to write $f(t)$ only in terms of $f$ and its derivatives at $t=0$?
I assume at minimum $M$ needs a connection, since otherwise I'm not sure how to even define the second and higher-order derivatives of $f$. With a metric one can define a "first-order approximation" of $f$ by
$$f(t) \approx \exp_{f(0)} \left[t f'(0)\right]$$
but what would the higher-order approximations look like?
 A: One natural way to obtain something like the Taylor expansion of a curve in the manifold is to probe it using a smooth function $\phi:M\to\mathbb{R}$ and expand the composition $\phi\circ f$. To that end, one may write down the Taylor expansion of $\phi\circ f$ in a local chart around $p=f(0)$, 
$$\phi(f(t))=\phi(p)+\partial_i\phi \dot f^i(0)\cdot t+\frac12\left((\partial_{ij}\phi)\dot f^i(0)\dot f^j(0)+\partial_i\phi\ddot f^i(0)\right)\cdot t^2+\dots$$
Whereas the first-order term is recognized as $d\phi_{p}(v)$, e.g. as the cotangent vector $d\phi_{p}$ acting on the velocity vector $v=\dot f(0)$, the second-order terms do not have a natural interpretation; neither the second derivative $\partial_{ij}\phi$ nor the 'acceleration' $\ddot f^i(t)$ are tensorial.
If we have a connection $\nabla$, we can rewrite the expansion as
$$\phi(f(t))=\phi(p)+d\phi_{p}(v)+\frac12\left(H\phi^{\nabla}_{p}(v,v)+d\phi_{p}(a)\right)\cdot t^2+\dots,$$
where $H\phi^{\nabla}_{p}$ is the covariant Hessian* of $\phi$ at $p$, which is evaluated on the velocity vector $v$ twice, and $a=\nabla_{\dot f(0)}\dot f$ is the covariant derivative of the velocity vector field along the curve in the direction of $\dot f(0)$ (covariant acceleration). 
Both second-order terms are now tensorial, i.e. natural with respect to pullback and pushforward by a smooth morphism $\psi:M\to N$ into a different manifold $N$ with connection $\nabla'$ (i.e. such that $\nabla$ and $\nabla'$ are compatible). Moreover, the parts belonging to the function $\phi$ are neatly separated from the parts belonging to the curve $f$: the former are encoded in covariant tensors while the latter occur as arguments of the covariant tensors, i.e. vectors. 
*Note that the Hessian is a bilinear form, and symmetric iff the torsion of $\nabla$ vanishes.
A: If $M$ is isometrically embedded into some Euclidean space then covariant differentiation (wrt to the induced Levi Civita connection) is nothing but differentiation wrt to an ambient Euclidean space and orthogonal projection onto the tangent space.
Using the Nash Embedding theorem this approach is probably the easiest way in case of (pseudo) Riemannian manifolds (ignoring for a moment the fact that the Nash embedding theorem is a rather deep and nontrivial result). Higher derivatives will be, nevertheless, an unpleasant challenge from the algebraic point of view.
A: Would the following work?
Take a (pseudo-) Riemannian manifold, with a tensor field $\mathbf T$ defined on some region containing a given point $p$. Associate points near $p$ with vectors $\mathbf v$ in the tangent space at $p$, via the exponential map. Intuitively, $\mathbf v$ gives the direction and magnitude of the point from $p$. Then the value of $\mathbf T$ at $\operatorname{exp}_p(\mathbf v)$ would be approximated by the Maclaurin series:
$$\mathbf T + \nabla_{\mathbf v}\mathbf T + \frac{1}{2!}\nabla_{\mathbf v}\nabla_{\mathbf v}\mathbf T  + \frac{1}{3!}\nabla_{\mathbf v}\nabla_{\mathbf v}\nabla_{\mathbf v}\mathbf T + \cdots.$$
All quantities are evaluated at $p$. $\nabla$ is the covariant derivative associated with the metric-compatible (Levi-Civita) connection, if this matters. The subscript in $\nabla_{\mathbf v}$ means differentiation in the direction of $\mathbf v$, which in components is: $v^\mu\nabla_\mu T^\cdots_\cdots \equiv T^\cdots_{\cdots\,;\mu}v^\mu$. Each term in the expansion above is a tensor of the same rank as $\mathbf T$.
Of course this will inherit the limitations of the ordinary Maclaurin series: it may not converge, or may converge to something other than the actual value of $\mathbf T$ at $\operatorname{exp}_p(\mathbf v)$. An additional issue is that geodesics may cross, so $\operatorname{exp}_p(\mathbf v) = \operatorname{exp}_p(\mathbf u)$ for $\mathbf v\ne\mathbf u$, and the map would not be uniquely defined there.
