There are many examples in which such a parallel transport does not exist (for example, take a non-constant function on the base $M$ and pull it back to $E$). However, it does exist in very specific cases, such as your example of a vector bundle with fiber metric. Let us try to generalize your example.
Let $E\to M$ be a fiber bundle with fiber $F$. So $E$ can be represented by a collection of trivializations and transition maps. The transition maps are all of the form $$\varphi_{\alpha,\beta}:U_\alpha\cap U_\beta\to \mathrm{diff}(F).$$ Let $G$ be a Lie subgroup of $\mathrm{diff}(F)$. A $G$-structure on our fiber bundle is a representation in which all the transition maps admit values in $G$.
Example: A vector bundle is merely a fiber bundle with fiber $\mathbb{R}^k$, equipped with a $GL_k(\mathbb{R})$-structure. Once you give this vector bundle a fiber metric, you actually reduce the structure group to $O_k(\mathbb{R})$.
We continue with a fiber bundle $E\to M$. To begin with, every fiber $E_p$ can be identified with the fiber model $F$ in many different ways. In fact, the set of diffeomorphisms $E_p\to F$ is diffeomorphic to $\mathrm{diff}(F)$. Once you equip $E$ with a $G$-structure, you distinguish a set of "special" identifications - those that respect the $G$-structure. The set of such identifications is diffeomorphic to $G$. (For example, if $E$ is a vector bundle, then the special identifications are the linear isomorphisms $E_p\to\mathbb{R}^k$. If you have a metric, the special identifications are the isometric linear isomorphisms).
Assume now that $E\to M$ has a $G$-structure and a compatible connection $\nabla$. That is, all the parallel transport maps of $\nabla$ respect the $G$-structure. Let $h:F\to\mathbb{R}$ be a function which respects the $G$-action on $F$. In other words, $h$ is constant on every $G$-orbit in $F$. Note that in this case, there is a well defined function $\tilde{h}:E\to\mathbb{R}$, which is induced by $h$. Namely, for $q\in E_p\subset E$ we have $$\tilde{h}(q)=h\circ\psi_p(q),$$where $\psi_p:E_p\to F$ is an identification that respects the $G$-structure. (Our assumptions guarantee that this is well defined). The conncection $\nabla$ will then satisfy what you want for the function $\tilde{h}$.
I leave it you to convince yourself that your example is a particular case of the situation described above.
Edit: Under mild further assumptions, the above story is in fact the only context in which this phenomenon occurs. Namely, let $E\to M$ be a fiber bundle equipped with a connection $\nabla$, and let $h:E\to\mathbb{R}$ be a function which is preserved by parallel transport of $\nabla$. We only add the assumption that parallel transport exists for every path (this is not always the case in general fiber bundles).
Choose a point $p\in M$, and let $G$ be the group of all diffeomorphisms $E_p\to E_p$ which respect $h|_{E_p}$. Then, construct a collection of trivializations of $E\to M$ using $E_p$ as the model fiber, where all identifications $E_{p'}\to E_p$ are obtained from parallel transport along some path. There are many different ways to do that, but all the transition maps will have values in $G$. This means that the function $h$ is of the sort described above.
Remark: Willie Wong's comment is correct and gives a very simple necessary and sufficient condition. Having said that, all the examples I have encountered of such behavior of a function and a connection originate naturally from a function on the model fiber, as described in this answer (this is definitely the case in your example). In other words, there is usually a bigger picture behind such a function.