Functional Derivative I am completely confused about calculating the (infinitesimal) change of an expression that depends on a function, when I vary the function. Consider the following:
$$
U(x;\rho)=\int_{0}^{x}\left[1-\int_{s}^{1}F(\rho(\xi))f(\xi)d\xi\right]^{n-1}ds
$$
where $F:[0,1]\rightarrow [0,1]$ is continuous and increasing; $f(x)=\frac{dF(x)}{dx}$, $x\in[0,1]$, and $n>2$. Fix the value of $x$. Then, the above expression becomes a functional (?). Now, I want to compute the change in $U$ for a fix $x$ when the function $\rho$ varies. According to me, this is what the resulting expression should look like:
$$
\frac{dU(\rho)}{d\rho}=(n-1)\int_{0}^{x}\left[1-\int_{s}^{1}F(\rho(\xi))f(\xi)d\xi\right]^{n-2}\left(\int_{s}^{1}f(\rho(\xi))f(\xi)d\xi\right)ds
$$
Can someone please tell me if what I'm doing is correct? My confusion stems from the fact that for a given value of $x$, the above expression becomes a functional and I am not really confident about differentiating functionals.
Any help is really, really appreciated!!!
 A: Obtaining the Fréchet derivative is straightforward, basically the same as 'ordinary' differentiation, except that more attention needs to be paid to the details. You have a slight omission in your formula above; while notationally bothersome, I think it is a good idea to remember that the derivative is a linear operator, and to actually apply the derivative to some point. So, rather than computing $Df(x)$, instead compute $Df(x)(h)$. In $\mathbb{R}^n$, this is of little consequence, but in 'function' spaces it matters.
To informally guess the derivative, I use the 'little o' notation and then check the details. However, for the details. I think it is easiest to use composition rule, ie, if $f,g$ are (Fréchet) differentiable with appropriate domains/ranges, then $D f \circ g (x) = D f(g(x)) Dg(x)$.
You didn't mention what space $\rho$ 'lives in', I will take it to be $X=L_{\infty}[0,1]$. Also, I am going to assume that $F$ is continuously differentiable on $[0,1]$.
Define the following functions:
$$\phi_1:X \to X, \ \ \phi_1(g)(t) = F(g(t))F'(t), \\
\phi_2:X \to X, \ \ \phi_2(g)(t) = \int_t^1 g(\tau) d\tau, \\
\phi_3:X \to X, \ \ \phi_3(g)(t) = (1-g(t))^{n-1}, \\
\phi_4:X \to \mathbb{R}, \ \ \phi_4(g)(t) = \int_0^x g(\tau) d\tau.$$
Then we have $U(x,\rho) = \phi_4(\phi_3(\phi_2(\phi_1(\rho))))$. I need to show that each is differentiable, and compute the derivative.
First, I need an estimate: Suppose $g:\mathbb{R} \to \mathbb{R}$ is continuously differentiable. Let $K \subset \mathbb{R}$ be compact, and $x \in K$. Then, for any $\delta$, Taylor's theorem gives:
$$g(x+\delta)-g(x)-g'(x)\delta = \int_0^1 (g'(x+t \delta)-g'(x))\, dt \, \delta.$$
Now, since $g'$ is continuous, it is uniformly continuous on any compact set. It is straightforward to show that the set $K_{\alpha} = \{ x+y | x \in K, |y| \leq \alpha \}$ is compact, for any $\alpha$. Now let $\epsilon > 0$, and choose $\alpha > 0$ small enough so that $|g'(x)-g'(y)| < \epsilon$ whenever $x,y \in K_{\alpha}$, and $|x-y| < \alpha$. Then, if $x \in K$, and $|\delta| < \alpha$, we have
$$|g(x+\delta)-g(x)-g'(x)\delta| < \epsilon |\delta|.$$
Consider $\phi_1$. Using the estimate, let $\epsilon >0$, and $K$ be a compact set. Then $\exists \alpha$ such that if $x \in K$, and $|\delta| < \alpha$, we have $|F(x+\delta)-F(x)-F'(x)\delta| < \epsilon |\delta|$. Also, note that $B=\sup_{t\in[0,1]} | F'(t)| < \infty$. Now suppose $g \in X$, then we have $|g(t)| \leq ||g||$ a.e. The set $K = \{x | |x| \leq ||g||\}$ is compact, so with $||\delta|| < \alpha$, we have the estimate:
$$|F(g(t)+\delta(t))F'(t)-F(g(t))F'(t)-F'(g(t))F'(t)\delta(t)| < \epsilon  B ||\delta||, \mbox{a.e.} \ t.$$
It follows (after checking for continuity, of course) that $D \phi_1(g)(\delta) (t) = F'(g(t))F'(t)\delta(t)$.
Thankfully $\phi_2$ is a little easier, it should be fairly clear that $D \phi_2(g)(\delta) (t) = \int_t^1 \delta(\tau) d \tau$.
$\phi_3$ follows the same lines as $\phi_1$, except applied to the function $x \mapsto (1-x)^{n-1}$. This yields $\phi_3(g)(\delta) (t) =-(n-1)(1-g(t))^{n-2} \delta(t)$.
Finally, $D \phi_4(g)(\delta) = \int_0^x \delta(\tau) d \tau$.
Now 'all' we need to do is compose the derivatives.
We have
$$D \phi_2 \circ \phi_1 (\rho) (\delta) (t) = D \phi_2 ( \phi_1 (\rho) ) D \phi_1 ( \rho) (\delta) (t) = \int_t^1 F'(\rho(\tau))F'(\tau) \delta(\tau) d \tau.$$
Next
$$D \phi_3 \circ \phi_2 \circ \phi_1 (\rho) (\delta) (t) = -(n-1)(1-\int_t^1 F(\rho(\tau))F'(\tau) d \tau)^{n-2} \int_t^1 F'(\rho(\tau))F'(\tau) \delta(\tau) d \tau.$$
And finally (really), we have
$$D \phi_4 \circ \phi_3 \circ \phi_2 \circ \phi_1 (\rho) (\delta) = -(n-1) \int_o^x (1-\int_t^1 F(\rho(\tau))F'(\tau) d \tau)^{n-2} \int_t^1 F'(\rho(\tau))F'(\tau) \delta(\tau) d \tau d t.$$
Notice how the 'perturbation' ($\delta$) works its way into the last integral. This was missing in your formula above.
A: The logic of computation looks okay to me. To alleviate your concerns about differentiating a functional, consider replacing $\rho$ with $\rho+h\psi$, where $\psi$ is another function (direction vector) and $h$ is a real variable. Holding both functions fixed, differentiate with respect to $h$ - this is the usual derivative of a real function. The result will be a linear functional of $\psi$, which is the derivative you are looking for. In particular, the derivative is zero if it is zero for every direction $\psi$ in your space of interest. 
