I think your statement is true on connected component of $\mathrm{Fix}(\phi)$, though I don't have a counter example for different components with different dimensions if $M$ is connected.
Suppose $\phi(x)=x$, and suppose its connected component in $\mathrm{Fix}(\phi)$ is not reduced to $\{x\}$. Let $v \in T_xM$ such that $\mathrm{d}\phi(x)v = v$. Let $\gamma$ be the unique geodesic with $\gamma(0) = x$ and $\gamma'(0) = v$. Then $\phi\circ \gamma$ is a geodesic (as $\phi$ maps geodesics to geodesics), with same initial data than $\gamma$. Thus, $\phi\circ \gamma = \gamma$.
Now, consider $H_x \subset T_xM$ the set of all fixed vectors of $\mathrm{d}\phi(x)$. It is a linear subspace. Let $U\subset T_xM$ be a convex neighbourhood of the origin on which the exponential map $\exp_x$ is a diffeomorphism onto its image. Suppose moreover that $\exp_x(U)$ is geodesically convex (one can assume this by reducing $U$). The idea is to prove that $\exp_x(U\cap H_x) = \mathrm{Fix}(\phi)\cap \exp_x(U)$. If this result is true, this shows that $\mathrm{Fix} (\phi)$ is a submanifold of $M$ with $H_x = T_x \mathrm{Fix}(\phi)$. Thus, $\dim H_x$ is locally constant as the rank of the tangent space of a submanifold.
The first inclusion $\exp_x(H_x\cap U) \subset \mathrm{Fix}(\phi)\cap \exp_x(U)$ is clear because of the first paragraph: the entire geodesic $\gamma$, with $\gamma(0) = 0$ and $\gamma'(0) = v$ is fixed by $\phi$, thus $\phi(\gamma(t)) = \gamma(t)$ for all $t$ and $\exp_x(tv) \in \mathrm{Fix}(\phi)$.
Conversly, suppose $p \in \mathrm{Fix}(\phi)\cap \exp_x(U)$. As $\exp_x(U)$ is geodesically convex, and $\exp_x$ is a diffeomorphism from $U$ to $\exp_x(U)$, let $\gamma$ be the unique minimizing geodesix from $x$ to $p$. Then $\phi\circ \gamma$ is also a minimizing geodesic from $x$ to $p$. By the assumptions on $U$, $\gamma = \phi\circ \gamma$, and as $\gamma(0)$ is a fixed point of $\phi$, this shows that $\gamma'(0)$ is fixed by $\mathrm{d}\phi(x)$ and $v \in H_x$.
We have shown that $\exp_x\left(H_x \cap U \right)$ is diffeomorphic to $\mathrm{Fix}(\phi) \cap \exp_x(U)$, and then it is a submanifold. The results follow.
Notice that we have shown more than that: we have shown that $\mathrm{Fix}(\phi)$ is a geodesic submanifold.