Functional derivative of the square of a functional How should I compute the functional derivative of the following functional:
$$
F(p(x)) = \left[\int\cos(x)p(x)dx - \int\cos(y)q(y)dy\right]^2
$$
I know that the functional derivative $\frac{\delta F}{\delta p(x)}$ is defined as
$$
\left[\frac{d}{d\epsilon}F(p +\epsilon\phi)\right]_{\epsilon=0} = \int \frac{\delta F}{\delta p(x)}\phi(x)dx
$$
but I am not sure how the chain rule is defined here in the case of a function of a functional, i.e., what's the functional derivative of $F = g(H(p(x)))$ where $g$ is a differentiable function and $H$ is a functional.
 A: The problem you found in the application of the formula for functional derivatives above is due the fact that it is not correct, since it is valid only for particular classes of functionals: the general definition of functional derivative is exactly the right side of the one you tried to use, i.e.
$$
\frac{\delta F}{\delta p} [p](\phi)\triangleq \left[\frac{\mathrm{d}}{\mathrm{d}\epsilon}F(p +\epsilon\phi)\right]_{\epsilon=0} = \lim_{\epsilon \to 0} \frac{F(p +\epsilon\phi) -F(p)}{\epsilon}\label{1}\tag{1}
$$
Said that, applying the definition to the quadratic functional under analysis we get
$$
\frac{\delta F}{\delta p} [p](\phi) = 2\left[\int\cos(x)p(x)\mathrm{d}x - \int\cos(y)q(y)\mathrm{d}y\right] \int\cos(x)\phi(x)\mathrm{d}x
$$
Edit following the comment of the Asker.
In order to clarify some of the doubts pointed out in the comment, I expanded a bit definition \eqref{1} above to analyze in a clearer way its meaning: it states that, in order to calculate the functional derivative of a functional $F$ at a "point" $p$, you should calculate the ordinary first order derivative of the functional applied to the variation $p+\epsilon \phi$, considered as a function of the real variable $\epsilon$. This implies that we should proceed in the calculations by using all the ordinary rules for differentiation including the chain rules: thus, when the functional $F$ is given by the composition of a ($C^\infty$-real valued) function of a real variable and a (real valued) functional $H$, i.e.
$$
F(p)=g\circ H(P)= g\big(H(p)\big),
$$
we have that
$$
\begin{split}
\frac{\delta F}{\delta p} [p](\phi) &= \left[\frac{\mathrm{d}}{\mathrm{d}\epsilon}g\big( H(p +\epsilon\phi)\big)\right]_{\epsilon=0} = \lim_{\epsilon \to 0} \frac{g\big( H(p +\epsilon\phi)\big) -g\big( H(p) \big)}{\epsilon}\\
& = \lim_{\epsilon \to 0} \frac{g\big( H(p +\epsilon\phi)\big) -g\big( H(p)\big)}{ {H(p +\epsilon\phi) -H(p)}}\cdot\frac{H(p +\epsilon\phi) - H(p)}{\epsilon}\\
&=\left. \frac{\mathrm{d}g(x)}{\mathrm{d} x}\right|_{x=H(p)}\cdot \left[\frac{\mathrm{d}}{\mathrm{d}\epsilon}H(p +\epsilon\phi)\right]_{\epsilon=0}\\
&=\left. \frac{\mathrm{d}g(x)}{\mathrm{d} x}\right|_{x=H(p)}\cdot \frac{\delta H}{\delta p} [p](\phi)
\end{split}
$$
which is basically an application (and the basic proof in disguise) of the chain rule formula.
Notes

*

*An analysis of the general definition of functional derivatives is offered in this Q&A, where you can have a look in order to have more information and some references on the concept.


*Regarding references: almost any book in the calculus of variation includes several examples of calculations of functional derivatives. However, they mostly (if not exclusively) deal with functional of integral type, which are not of the same kind of the functional studied here. Therefore my advice is to have a look at this, this, this and this Q&A which deal with functionals of non integral type. And, apart from consult the references given in the cited Q&A if needed, I advice to look at the first volume [1] of the monograph by Mariano Giaquinta and Stefan Hildebrandt, in my opinion a wonderful reference.
[1] Giaquinta, Mariano; Hildebrandt, Stefan (1996), Calculus of Variations I. The Lagrangian Formalism, Grundlehren der Mathematischen Wissenschaften, 310 (1st ed.), Berlin: Springer–Verlag, pp. xxix+475, ISBN 3-540-50625-X, MR 1368401, Zbl 0853.49001.
