Given a Riemannian manifold $(M,g)$, the paths of a Brownian motion on it can be written as the following stochastic differential equation in local coordinates: $$ dX_t = \sqrt{g^{-1}} dB_t - \frac{1}{2} g^{ij}\Gamma^k_{ij} dt = \sigma(X_t)\, dB_t + \vec{b}(X_t) \,dt $$ where $B_t$ is an $n$ dimensional Wiener process and $g_{ij}\sigma^i_k\sigma^j_\ell=\delta_{k\ell}$.
My question is conceptual and geometric in nature: how can there be a drift term in this equation?
Algebraically, I understand, roughly speaking, that it arises from the extra temporal terms in Ito's formula. However, in a general relativity sort of way, one can consider $g$ to be a "warping of space" (say for $M=\mathbb{R}^n$), and we note $g$ is always symmetric. The metric does not depend on directions, but only on locations (unlike say for Finsler manifolds). In other words, acceleration in one direction due to the curvature also occurs in the opposite direction, meaning the effect of the curvature on the diffusion is also symmetric. So, geometrically, how can $\vec{b}$ exist, as it by definition favors some particular direction?
This is even weirder to me when I think about Riemannian normal coordinates (say at $p$), where $g_{ij}=\delta_{ij}$ and thus $\Gamma^k_{ij}=0$ at $p$. Thus, $\vec{b}=0$ at $p$, in that system. One can do this at every point. I suppose the drift would not disappear off of $p$, but it still seems odd to me that the presence of drift would not somehow be an invariant. Undoubtedly, I am missing something here.
I think something to do with the heat equation generating the SDE above, i.e. $\partial_t u = \Delta_g u/2$, may be useful.
Edit: It's useful to note that the term "drops out" of the equation for the Laplace-Beltrami operator in local coordinates (see the cross-posted version of this question on MathOverflow).