Calculus of Variations Problem with Convolution Constraint Consider the following problem:
\begin{eqnarray}
&\inf_{f} {\int\limits_0^1 g(x)f(x)   dx}\\
&\textrm{s.t.} \quad \int\limits_0^{2} g(x)(f\ast f(x))   dx=c\\
& \quad f:[0,1] \rightarrow [0,1]
\end{eqnarray}
where $f\ast f(x)$ is the convolution of $f$ with itself, $c>0$ is low enough so that the feasible set is non-empty.
I plan to start working with the first variation, but wanted to pause and check here if there is a standard method of dealing with this kind of problem (maybe convolution suggests a move to the frequency domain?). Thank you.
Note: Original problem had $g(x)=xe^{-x}$, but the solution is more general.
 A: The first variation needs two perturbation $\eta_1$ and $\eta_2$ and they are dependent on each other. In effect this gives the Lagrange multiplier.
Define $g(x):=xe^{-x}$. In fact, it will be clear from the derivation below that this function is a red herring so long as it is not zero. We have either
$$\label{conVar}\int g(x)(f*\eta)(x)\,dx=0 \tag1$$
or the stationary function $f$ satisfies
$$\label{var}\int g(x)(\eta(x)+\lambda f*\eta(x))\,dx=0,\tag2$$
for all admissible $\eta$, where $\lambda$ is some constant, i.e., the Lagrange multiplier. Denote the Fourier transform of $f$ by $\mathcal F[f]$ and the complex conjugation of $g$ by $\bar g$. By the Plancherel theorem, we have for Equation \eqref{conVar}
\begin{align}
0&=\int g(x)(f*\eta)(x))\,dx \\
&=\int \overline{\mathcal F[g]}(k)\,\mathcal F[f*\eta](k)\,dk \\
&=\int \overline{\mathcal F[g]}(k)\,\mathcal F[f](k)\mathcal F[\eta](k)\,dk
\end{align}
for all admissible $\eta$. So
$$\mathcal F[f]=0 \Longleftrightarrow f=0.$$
Depending on whether $c=0$ or not, $f=0$ is a stationary function or not.
By the same token, Equation \eqref{var} gives
\begin{align}
0&=\int g(\eta(x)+\lambda f*\eta(x))\,dx \\
&=\int g\big((\delta+\lambda f)*\eta(x)\big)\,dx \\
&=\int \overline{\mathcal F[g]}(k)\,\mathcal F[(\delta+\lambda f)*\eta(x)](k)\,dk \\
&= \int \overline{\mathcal F[g]}(k)\,\mathcal F[(\delta+\lambda f)](k) \,\mathcal F[\eta(x)](k)\,dk \\
&= \int \overline{\mathcal F[g]}(k)\,(1+\lambda\mathcal F[f])(k) \,\mathcal F[\eta](k)\,dk
\end{align}
for every admissible $\eta$. Since $\eta$ is arbitrary,
$$1+\lambda\mathcal F[f]=0 \Longleftrightarrow f(x)=-\frac1\lambda\delta(x).$$
$\therefore f:[0,1] \not\rightarrow [0,1].$
