Finding extremals of a functional with integral function as integrand Assume you want to find the extremals for the functional
$$
y \rightarrow \int_a^by(x)\left[\int_a^xy(\xi)\, d\xi\right]\, dx
$$
where $[a,b]\subset \mathbb{R}$ and $y\in \mathcal{C}^1\left([a,b],\mathbb{R}\right)$.
How can you write down the Euler Lagrange equations associated with this problem?
 A: So the functional to be minimized is
$$\int_a^by(x)\left[\int_a^xy(\xi)\, d\xi\right]dx.$$
I think I would "shift the perspective" a bit from $y$ to $\displaystyle\int y(x)\,dx$. (This is daw's suggestion in the comments above.) That is, let
$$Y(x)=\int_a^x y(\xi)\,d\xi.$$
Then the functional can be written as 
$$\int_a^bY(x) Y'(x)\,dx.$$
Now we let $L=YY',$ and the E-L equations are
$$\frac{\partial L}{\partial Y}-\frac{d}{dx}\frac{\partial L}{\partial Y'}=Y'-\frac{d}{dx}Y=Y'-Y'=0. $$
This procedure at least allows you to write down the E-L equations, even though, in this case, they tell you nothing. I suspect the minimization problem is still ill-defined.
Actually, we can say something more in this particular case:
$$\int_a^b Y(x)Y'(x)\,dx=\frac{Y^2(x)}{2}\Bigg|_a^b=\frac{Y^2(b)}{2}-\frac{Y^2(a)}{2}=\frac{Y^2(b)}{2},$$
by the definition of $Y$.
Minimizing this is tantamount to forcing $Y(b)=0,$ or as close to it as is feasible.
A: *

*OP's functional is non-negative
$$ F[y]~:=~\int_{[a,b]} \!\mathrm{d}x~y(x)\int_{[a,x]}\!\mathrm{d}\xi ~y(\xi)
~=~\iint_{[a,b]^2} \!\mathrm{d}x~\mathrm{d}\xi ~\theta(x\!-\!\xi) ~y(x)y(\xi)$$
$$~=~\frac{1}{2}\iint_{[a,b]^2} \!\mathrm{d}x~\mathrm{d}\xi ~y(x)y(\xi)
~\stackrel{(2)}{=}~\frac{G[y]^2}{2} ~\geq~ 0,\tag{1}$$
where 
$$G[y]~:=~\int_{[a,b]} \!\mathrm{d}x~y(x).\tag{2}$$

*The functional/variational derivative is
$$\frac{\delta F[y]}{\delta y(x)}~\stackrel{(1)}{=}~G[y]\frac{\delta G[y]}{\delta y(x)} ~\stackrel{(2)}{=}~G[y]1_{[a,b]}(x),\tag{3} $$
which is a more general notion than the Euler-Lagrange derivative.

*OP's sought-for equation is the vanishing of the functional derivative (3). Evidently, a stationary configuration $y:[a,b]\to \mathbb{R}$ has 
$$G[y]~\stackrel{(3)}{=}~0,\tag{4}$$
which is clearly a minimum (and in particular an extremum) for OP's functional (1).
