Does parallel transport change the subspace? Let $M$ be a Riemannian manifold and $N$ be an immersed submanifold. Take $\gamma$ a geodesic starting at and perpendicularly to $N$. Let $X(t)$ be a vector field along $\gamma$ such that $X(0) \in T_{\gamma(0)}N$. Assume that $X(t)$ is not necessarily parallel along $\gamma$. Take an instant $t_0$ and consider $X(t_0)$. Denote by $E_X(t)$ the parallel transport of $X(t_0)$ along $\gamma$. Does $E_X(0) \in T_{\gamma(0)}N$?
 A: Consider the plane $\{(x, y, 0)\mid x,y \in \Bbb R\}\subset\mathbb{R}^3$, the line $t\mapsto (0, 0, t)$ and the vector field $t\mapsto(1, 1, t)$ along it. Then the parallel transport of no vector in the field along the curve will be tangent to the plane.
A: Negative answer. Counter-example: $(M,g) = (\Bbb R^3, \langle\cdot, \cdot \rangle)$, $N = \Bbb R^2\times \{0\}$, $\gamma(t) = (0,0,t)$ and $X(t) = (0,0,\sin t)$. Then $X(0) = 0 \in T_{\gamma(0)}N$, but the parallel transport of $X(\pi/2) = (0,0,1)$ from $\gamma(\pi/2) = (0,0,\pi/2)$ to $\gamma(0) =0$ via $\gamma$ is $(0,0,1) \not\in T_{\gamma(0)}N$. The values $X(t)$ for $t\neq t_0$ should not influence the result of transporting $X(t_0)$, since a priori one transports vectors and not fields.
A: We of course understand that $\dot \gamma(0) \ne 0$, lest $\gamma(t)$ be a rather trivial geodesic.
We observe that, since $\dot \gamma(0)$ is "perpendicular" to $N$, the condition 
$\langle \dot \gamma(0), X(0) \rangle = 0 \tag 0$
is necessary, but not sufficient, for
$X(0) \in T_{\gamma(0)}N, \tag 1$
for (0) says that $X(0)$ has no component along $\dot \gamma(0)$; in the event that $N$ is a hypersurface in $M$, that is,
$\dim N = \dim M - 1, \tag 2$
(0) is in fact also sufficient, for then $\dot \gamma(0)$ and $T_{\gamma(0)}N$ span $T_{\gamma}(0)N$; in fact, (2) implies
$T_{\gamma(0)}M = \langle \dot \gamma(0) \rangle \oplus T_{\gamma(0)}N, \tag 3$
where $\langle \dot \gamma(0) \rangle$ is the one-dimensional subspace of $T_{\gamma(0)}M$ generated by $\dot \gamma(0)$.
Now consider any vector field $X(t)$ along $\gamma(t)$; we have
$\nabla_{\dot \gamma} \langle \dot \gamma, X \rangle = \langle \nabla_{\dot \gamma} \dot \gamma, X \rangle + \langle \dot \gamma, \nabla_{\dot \gamma} X \rangle; \tag 4$
if $X(t)$ is parallel along $\gamma(t)$, then
$\nabla_{\dot \gamma}X = 0; \tag 5$
thus (4) becomes
$\nabla_{\dot \gamma} \langle \dot \gamma, X \rangle = \langle \nabla_{\dot \gamma} \dot \gamma, X \rangle; \tag 6$
now the general formulation of the geodesic equation as I understand it is
$\nabla_{\dot \gamma} \dot \gamma = f(t) \dot \gamma; \tag 7$
here $f(t)$ is determined by the choice of the running parameter $t$ of $\gamma(t)$; if we substitute this into (6), we obtain
$\nabla_{\dot \gamma} \langle \dot \gamma, X \rangle = f(t) \langle  \dot \gamma, X \rangle; \tag 8$
the solution to this linear, first order differential equation for $\langle \dot \gamma, X \rangle$ with $\langle \dot \gamma(t_0), X(t_0) \rangle = \langle \dot \gamma, X \rangle(t_0)$ is
$\langle \dot \gamma, X \rangle(t) = \exp \left (\displaystyle \int_{t_0}^t f(s) \; ds \right ) \langle \dot \gamma, X \rangle (t_0); \tag 9$ 
since
$\forall t \in \Bbb R, \;\exp \left (\displaystyle \int_{t_0}^t f(s) \; ds \right ) \ne 0, \tag{10}$
(9) shows that
$\langle \dot \gamma(t), X(t) \rangle = \langle \dot \gamma, X \rangle(t) = 0 \Longleftrightarrow  \langle \dot \gamma, X \rangle (t_0) = \langle \dot \gamma(t_0), X(t_0) \rangle = 0; \tag{11}$
of course, if $\gamma(t)$ is affinely parametrized then
$f(t) = 0, \tag{12}$
and we in fact have
$\langle \dot \gamma, X \rangle(t) = \langle \dot \gamma, X \rangle (t_0), \; \forall t \in \Bbb R; \tag{13}$
now taking $t_0 = 0$ yields
$\langle \dot \gamma(t), X(t) \rangle = 0 \Longleftrightarrow \langle \dot \gamma(0), X(0) \rangle = 0; \tag{14}$
if we now choose $X(t)$ such that
$\langle \dot \gamma(t), X(t) \rangle \ne 0, \tag{15}$
(14) together with (0) shows the parallel transport of $X(t)$ along $\gamma(t)$ cannot satisfy
$X(0) \in T_{\gamma(0)} N, \tag{15}$
since it has a component in the direction $\dot \gamma(0)$.
