According to my computations, the function which minimizes $\int_\Omega \|\operatorname{curl} f\|^2~dx$ should satisfy $\operatorname{curl}(\operatorname{curl}(f)) = 0$ everywhere on $\Omega$, provided $\operatorname{curl} f = 0$ on $\partial \Omega$.
I followed the same kind of computation that the one demonstrating the the argmin of $\int_\Omega \|\nabla f\|^2~dx$ should satisfy $\Delta f = 0$. However, I am not sure whether my computations are right... May anyone check that please ?
1) We first start with a functional $G(f) = \int_\Omega \|\operatorname{curl} f\|^2~dx$.
2) We compute $V(f,h) = lim_{\epsilon\rightarrow 0} \frac{G(f+\epsilon h)-G(f)}{\epsilon} = 2\int_\Omega \operatorname{curl} f\cdot\operatorname{curl} h~dx$
3) We use the identity : $\operatorname{div}(A\times B) = -A\cdot\operatorname{curl} B + B\cdot\operatorname{curl} A$, with $A=\operatorname{curl} f$ and $B=h$
4) We obtain $\int_\Omega \operatorname{curl} f\cdot \operatorname{curl} h\:dx = -\int_\Omega \operatorname{div}(\operatorname{curl} f\times h)~dx + \int_\Omega h\cdot \operatorname{curl}(\operatorname{curl}(f))~dx$
5) We use the divergence theorem to obtain : $\int_\Omega \operatorname{curl} f\cdot \operatorname{curl} h~dx = -\int_{\partial\Omega} \operatorname{curl} f\times h~ds + \int_\Omega h\cdot\operatorname{curl}(\operatorname{curl}(f))~dx$
6) We assumed $\operatorname{curl} f = 0$ on $\partial\Omega$, so the first term is zero.
7) $V(f,h)$ should equal zero for all $h$ for the function to be minimized, so $\int_\Omega h\cdot\operatorname{curl}(\operatorname{curl}(f)) dx = 0~~\forall h$, which implies $\operatorname{curl}(\operatorname{curl}(f)) = 0$ locally.
I guess this reasonning may be wrong in several places..... or is it right?? Thanks!
