A diffeomorphism which maps geodesics to geodesics preserves the connection?

Question

Let $(M,\nabla^M),(N,\nabla^N)$ be two smooth manifolds with given (affine) connections on their (tangent bundles). We say a diffeomorphism ,$\phi:(M,\nabla^M)\rightarrow(N,\nabla^N)$ is an isomorphism if: $\nabla^N_X{Y}=\phi_* \left( \nabla^{M}_{\phi^{-1}_*(X)} {\phi^{-1}_*(Y)} \right) \forall X,Y \in \Gamma(TN)$,

where the pushforward $\phi_*(X)(q)=d\phi_{\phi^{-1}(q)}[X \left(\phi^{-1}(q)\right)]$ is the corresponding isomoprhism of Lie algebras.

Assume $(M,\nabla)$ is a smooth manifold with an affine connection (on its tangent bundle). Let $\phi \in \text{Diff(M)}$. If for every geodesic $\gamma$ (w.r.t $\nabla$) $\phi \circ \gamma$ is a geodesic, is it true that it must be an isomorphism of $(M,\nabla)$?

(Note that if we ask this question with two connections $\nabla_1,\nabla_2$ , and assume that $\phi$ maps geodesics $\nabla_1$ into geodesics of $\nabla_2$ , then clearly the answer is negative. For example, we can take two different connections with identical geodesics, and $\phi = Id$)

Answer 1

No, it's not true.

Two connections $\nabla_1$ and $\nabla_2$ have the same geodesics if and only if their difference tensor $D(X,Y) = (\nabla_1)_X Y - (\nabla_2)_X Y$ is antisymmetric, meaning $D(X,Y) = - D(Y,X)$ for all $X,Y$. Let $\overline\nabla$ denote the Euclidean connection on $\mathbb R^3$, and choose a smooth bump function $\psi$ supported in a compact neighborhood of the origin. Define a new connection $\nabla$ on $\mathbb R^3$ by $$ \nabla_X Y = \overline\nabla_X Y + \psi X\times Y, $$ where $X\times Y$ is the usual cross product. Then the difference tensor between $\nabla$ and $\overline\nabla$ is $\psi X\times Y$, which is antisymmetric, so the geodesics for both connections are constant-speed straight lines.

Now let $\phi\colon \mathbb R^3\to \mathbb R^3$ be a translation. It takes straight lines to straight lines, so it takes $\nabla$-geodesics to $\nabla$-geodesics. But the pullback connection is $$ (\phi^*\nabla)_X Y = \overline\nabla_X Y + (\psi\circ\phi) X\times Y, $$ which is not the same as $\nabla$.