# Geodesic convexity and the 2nd fundamental form

Let $(M,g)$ be a Riemannian manifold, $\Omega\subset M$ be a closed set with smooth boundary $\partial\Omega$ and $\nu$ be the unit normal of $\partial\Omega$ pointing into $\Omega$.

$\Omega$ is said to be geodesically convex iff $\forall x_0, x_1\in\Omega$ $\exists c:[0,1]\stackrel{\text{geodesic}}\to(M,g)$ s.t. $c(0)=x_0, c(1)=x_1$, $c([0,1])\subset\Omega$, $\mathrm{Length}[c]=d_g(x_0,x_1)$.

Suppose $\Omega$ is geodesically convex. Then...

[Q.1] Does it hold that the 2nd fundamental form of $\partial\Omega$ toward $\nu$ is nonnegative definite at each point on $\partial\Omega$?

[Q.2] Let $\psi_r(x):=\mathrm{exp}^g_x [r\nu(x)]\in N$ $(x\in\partial\Omega)$. Then for small $|r|$, $\psi_r$ is an embedding. Here, does it hold that the inner 2nd fundamental form of $\psi_r$ is nonnegative definite at each point on $\partial\Omega$ when $r>0$ and is sufficinetly small?

Thank you.

I will address your first question. First define, for a fixed point $$p\in\partial \Omega$$, three conditions on $$\partial \Omega$$.

a) There is an open subset $$U\subset M$$ with $$p\in U$$, such that any two points in $$U\cap \Omega$$ can be joined by length minimising geodesic $$c:[0,1]\rightarrow M$$ with $$c[0,1]\subset U\cap \Omega$$.

b) Any geodesic $$c:(-\epsilon,0]\rightarrow M$$ with $$c(-\epsilon,0)\subset \mathrm{int(\Omega)}$$ and $$c(0)= p$$ hits the boundary transversally, i.e. $$g(\dot c(0),\nu(p))\neq 0$$.

c) The second fundamental form $$l_\nu(\cdot,\cdot)=g(\nabla_\cdot \cdot, \nu)$$ is non-negative at $$p$$.

We will prove that a) $$\Rightarrow$$ b) $$\Rightarrow$$ c), which answers your first question.

Step 1 Take geodesic normal coordinates $$x^1,\dots,x^n$$ , centred at $$p$$, with $$\partial_n\vert_p=\nu(p)$$. Using the implicit function theorem, one can show that there is a smooth function $$f:\mathbb{R}^{n-1}\rightarrow \mathbb{R}$$ such that $$\Omega \cap V =\{x^n\ge f(x^1,\dots,x^{n-1})\},$$ where $$V$$ denotes the coordinate patch. Next define smooth vector fields $$X_1,\dots,X_{n-1}$$ on $$V$$ by $$X_j(q)=\partial_j\vert_q + \partial_jf(x^1(q),\dots,x^{n-1}(q))\partial_n\vert_q.$$ If $$q\in V \cap \partial \Omega$$, then $$X_j(q) \in T_q\partial \Omega$$ and hence for $$1\le j\le n-1$$ we have $$\partial_jf(0) = X_j^k(p) g_{kl}(p) \nu^l(p)=g(X_j(p),\nu(p)) = 0 \tag{1}.$$

Step 2 We claim that $$\text{b)} \quad \Leftrightarrow \quad f \text{ has a local minimum at 0.} \tag{2}$$ Note that in the coordinates fixed above, b) means that $$\{x^n=0\}$$ (the hypersurface spanned by geodesics through $$p$$ which are not transversal) does not intersect $$int(\Omega)\cap V =\{x^n> f(x^1,\dots,x^{n-1})\}$$ near $$p$$, or in other words that $$f(x^1,\dots,x^{n-1})\ge 0$$ in a neighbourhood of $$p$$. This proves (2).

Step 3 We're now in a position to prove a) $$\Rightarrow$$ b). Suppose b) is wrong, then $$f$$ does not have a local minimum at $$0$$. I.e. for any neighbourhood $$U$$ of $$p$$ there is a point $$q\in \partial \Omega \cap U$$ such that $$x^n(q)=f(x^1(q),\dots,x^{n-1}(q))<0. \tag{3}$$ Since we are in geodesic coordinates, the unique minimising geodesic joining $$p$$ and $$q$$ is given by $$L =\{x^j=tx^j(q): 0\le t \le 1\}$$. Assuming a), we must have $$L\subset \Omega \cap U$$, which implies $$tx^n(q) \ge f(tx^1(q),\dots,tx^{n-1}(q)).$$ Divide by $$t$$ and take the limit $$t\rightarrow 0$$, then the right hand side will converge to $$0$$, since $$Df(0)=0$$ (see $$(1)$$). We obtain $$x^n(q)\ge0$$, which is a contradiction to $$(3)$$.

Step 4 We want to relate the second fundamental form to the Hessian of $$f$$. To this end note that, for $$1\le j \le n-1$$ and $$1\le k \le n$$ we have $$\nabla_kX_j (p)= (\Gamma_{kj}^l \partial_l + \partial_k \partial_j f \partial_n + \Gamma_{kn}^r \partial_r)(p) = \partial_j\partial_kf(p) \cdot \nu(p)$$ and since $$f$$ is independent of $$x^n$$, we further obtain $$\nabla_{X_k}X_j(p) = \partial_j\partial_kf(p) \cdot \nu(p),$$ which implies $$l_{\nu}(X_k,X_j)\vert _p = \partial_j\partial_kf(p)$$. Since the $$X_j$$ form a basis of $$T_p\partial \Omega$$ we have $$\text{c)}\quad \Leftrightarrow \quad f \text{ has non-negative Hessian at 0}. \tag{4}$$ From $$(2)$$ and $$(4)$$ it is evident that b) implies c).

Remark: With the same kind of argument one obtains an interesting result in the other direction. Assuming that the second fundamental form of $$\partial \Omega$$ is strictly positive, all geodesics coming from $$\Omega$$ hit $$\partial \Omega$$ transversally.