Set $d=\inf_{s\in S} d(v,s)$ to be the distance from $v$ to $S$. We show that finding the nearest point to $v$ lying within $S$ gives the solution.
Lemma 1 There exists $s\in S$ such that $\lVert v-s\rVert = d$ if we additionally assume $S$ complete.
Lemma 2 $t=v-s$ is orthogonal to $S$.
Lemma 3 This decomposition is unique.
Proof of 1 Take $d(s_n,v)\le d^2+\frac 1 n$. We show $s_n$ is Cauchy. Recall the parallelogram law $\lVert v_1+v_2\rVert^2+\lVert v_1-v_2\rVert^2=2(\lVert v_1\rVert^2+\lVert v_2\rVert^2)$. Set $v_1=v-s_n,v_2=v-s_m$. One finds $$\lVert s_m-s_n\rVert^2\le 2/m+2/n$$ and the result follows.
Proof of 2 Let $s'\in S$. $$\lVert v-(s+\lambda s')\rVert^2=\lVert v-s\rVert^2-2 \mathrm{Re}[\left<v-s,s'\right>]\lambda+\lVert s'\rVert^2 \lambda^2$$ but since $s$ minimized the length in question, considering small lambda the linear term must vanish. Hence $v-s$ is orthogonal to $S$.
Proof of 3 $v=s'+t'=s+t$ implies $(s'-s)+(t'-t)=0$. Taking inner products with each term shows both are zero.
As pointed out, for incomplete $S$ this fails. Just choose any situation like that of the finite sequences in $\ell^2$ where the natural 'orthogonal' terms form a series converging to a missing point in $S$.