Is the shortest distance between a point and a curve always normal to the curve? Suppose we have some point $P$ and some differentiable function $f$. Then let $P_C$ be a point on $f$ such that $|P-P_C|\le |P-f(t)|$ for any $t$. In other words, $P_C$ is one of the points closest to $P$ and lies on $f$. Intuitively, it seems to me that the line normal to $f$ at $P_C$ will always intersect $P$. However, I don't know of a proof for this. Is my intuition correct? Is there a proof of this statement? Is this true for all dimensions? If it's not true for all differentiable functions, is there a subset of functions for which it is true?
 A: Since there seems to be some misconceptions here, let me elaborate a bit on my comment. If $M$ is a $k$-dimensional submanifold (a subset that is locally smoothly equivalent to a ball in $\Bbb R^k$) of $\Bbb R^n$, then it has a $k$-dimensional tangent space at the point $p\in M$, often denoted $T_pM$. Orthogonal to this $k$-dimensional subspace is the $(n-k)$-dimensional space called the normal space of $M$ at $p$, often denoted $N_pM$. Given a point $Q\in\Bbb R^n$, if the point $p\in M$ is closest to (or, indeed, farthest from) $Q$, then essentially the calculation that @QuangHoang did will establish that the vector $Q-p$ is orthogonal to $T_pM$, i.e., is an element of the normal space $N_pM$. (You can think of this intuitively as an infinitesimal Pythagorean Theorem.)
The proof, as I suggested, is to parametrize a neighborhood $U$ of $p\in M$ by a ball $B$ in $\Bbb R^k$ by $\varphi\colon B\to U\subset\Bbb R^n$. Let's assume $\varphi(0)=p$.  Now consider the function
$$F\colon B\to\Bbb R, \quad F(s) = \|\varphi(s)-Q\|^2.$$
If $F$ has a (local) extreme point at $0$, then the chain rule tells us that for any vector $v\in\Bbb R^k$,
$$DF(0)v = 2 (\varphi(0)-Q)\cdot D\varphi(0)v = 0,$$
which says that $\varphi(0)-Q = p-Q$ is orthogonal to the image of $D\varphi(0)$. That image is the tangent space $T_pM$.
A: Corner cases aside, the distance attains minimum/maximum if and only if the square function also attains minimum/maximum
$$g(t) = d^2(t) = \|P-f(t)\|^2$$
On the other hand,
$$\frac{d}{dt}g(t) = 2(f(t)-P)\cdot f'(t)$$
and the statement follows.
