Why Gauss Lemma means the boundary of normal ball is a hypersurface? Pictures below is from the do Carmo's Riemannian Geometry, I don't know why  Gauss Lemma means the boundary of normal ball is a hypersurface, although it is very visualized. Besides, seemly, there is not explicit definition of hypersurface in this book.
Last, the hypersuface orthogonal to the geodesics start from $p$ is also  visualized in my view, but I also how to get it. Seemly, it is the process of Gauss lemma what make this point.






 A: Do Carmo's formulation is ambiguous, and he does not mean that the Gauss Lemma implies that the boundary of a geodesic ball is an hypersurface. The author might have prefered this formulation:

The boundary of a geodesic ball, contained in a normal neighbourhood, is an hypersurface, and by Gauss Lemma, it is normal to geodesic rays starting from $p$.

The fact that small geodesic balls are hypersurfaces comes from the very definition of a normal neighbourhood. Indeed, for $V\subset T_pM$, being a normal neighbourhood of $0 \in T_pM$ means that $\exp_p \colon V \to \exp_p(V) \subset M$ is a diffeomorphism. Thus, as a sphere of radius $\varepsilon$ in $T_pM$ is an hypersurface and for $\varepsilon$ small enough, it is contained in $V$, this imply that its image is a compact hypersurface of $\exp_p(V)$, which is open in $M$. It follows that it is an hypersurface of $M$.
The fact that it is normal to geodesic rays is a direct consequence of Gauss Lemma. If $v \in T_pM$ is a non-zero vector and if $w \perp v$, then, by Gauss Lemma:
$$
\forall t, \langle \mathrm{d}\exp_p(tv)v,\mathrm{d}\exp_p(tv)w\rangle = \langle v,w\rangle = 0
$$
and therefore, $\mathrm{d}\exp_p(tv)w$ is orthogonal to $\mathrm{d}\exp_p(tv)v$, which is the tangent vector of $\gamma(t) = \exp_p(tv)$.
Comment the other answer claims a wrong statement. It is false that the exponential map is an immersion. For example, on the unit sphere, if $v$ and $w$ are orthonormal vector at $p \in S^n$, then $\mathrm{d}\exp_p(\pi v) w = 0$ by easy Jacobi fields analysis: indeed $\mathrm{d}\exp_p(tv)(tw) = \sin t W$ where $W$ is the parallel transport of $w$ along $\exp(tv)$. Therefore, $\mathrm{d}\exp_p(\pi v)$ is not injective. However, it is true that in non-positive curvature, it is.
A: Heuristically Gauss' lemma allows us to show that the exponential map is an immersion. Indeed if $v$, $w$ belong to $T_{p}M$, then
$$ \langle d\exp_{p}(v-w),d\exp_{p}(v-w)\rangle = \langle v-w, v-w\rangle=
\|v-w\|^{2}.$$
If $d\exp_{p}v=d\exp_{p}w$, then it follows that $\|v-w\|^{2}=0$, and so $v=w$. This proves that $d\exp_{p}$ is an injection, thus $\exp_{p}$ is an immersion.
A sphere in $T_{p}M$ is a smooth submanifold of codimension $1$ in $T_{p}M$. Then the local immersion property of the exponential map implies that it is a local diffeomorphism, so the image of the sphere under the exponential map is still locally a hypersurface, and is thus an immersed submanifold of $M$.
This is the best of my understanding, but I could be wrong. Without much details it is difficult to come up with a correct explanation....
