Maximum of $ F(f)=\int_0^1 |f(x)|^2\; dx-\left(\int_0^1 f(x)\; dx\right)^2 $ over a subset of continuous functions on $[0,1]$ 
Let $X$ be a subset of $C([0,1])$ with
$$
X=\big\{f\in C([0,1]): 0\le f(x)\le x,\text{$f$ is a polynomial}\big\}
$$
where $C([0,1])$ denotes the space of continuous real-valued functions on $[0,1]$. Define a nonlinear functional $F$ on $C([0,1])$ as
$$
F(f)=\int_0^1 |f(x)|^2\; dx-\left(\int_0^1 f(x)\; dx\right)^2
$$
Can $F$ achieve a maximum in the set $X$?


Remarks.

*

*If one restricts the set to the linear functions $f$ of the form $f(x)=kx$ with $0\le k\le 1$, then this reduces to a min/max problem for values of $k$:
$$
F(f) = \int_0^1 (kx)^2\;dx-\left(\int_0^1kx\;dx\right)^2=\frac{k^2}{3}-\frac{k^2}{4} = \frac{k^2}{12}
$$
So the maximum is achieved at $f(x)=x$ for the restriction.


*One could also restrict the functions to be parabolas of the form $f(x)=kx^2$ with $k\in[0,1]$, and get
$$
F(f)=\frac{k^2}{5}-\frac{k^2}{9}=\frac{4k^2}{45},
$$
where the maximum is achieved at $f(x)=x^2$.


*If one allows one more parameter and consider $f(x)=kx^m$, then
$$
F(f)=k^2\cdot \left(\frac{1}{2m+1}-\frac{1}{(m+1)^2}\right)=\frac{k^2m^2}{(2m+1)(m+1)^2}\;.
$$
Then one can study the maximum of $g(m)=\frac{m^2}{(2m+1)(m+1)^2}$ over positive integers $m$.
But this is far from analyzing all the polynomials in $X$.


*Another observation is that the functional can be written as $F(f)=\|f\|_{L^2}^2-\|f\|_{L^1}^2$. But this seems not helpful at all.


*This question was motivated by this unanswered question on the site. I was curious and attempted to solve that one but I didn't get anything. So I restrict the attention to the set of polynomials here.
 A: For the sake of convenience, we will consider the functional
$$ \mathsf{Var}(f) = \int_{0}^{1} f(x)^2 \, \mathrm{d}x - \left( \int_{0}^{1} f(x) \, \mathrm{d}x \right)^2 $$
on the set $\overline{X}$ of all measurable functions $f$ on $[0, 1]$ such that $0 \leq f(x) \leq x$ for all $x \in [0, 1]$. Note that $\overline{X}$ is the closure of OP's set $X$ in $L^2([0,1])$, justifying the notation.
In order to find the maximum of $\mathsf{Var}(\cdot)$, for each $c \in [0, 1]$ we define $g_c : [0, 1] \to \mathbb{R}$ by
$$ g_c(x) = x \mathbf{1}_{[c,1]}(x) = \begin{cases}
x, & x \geq c, \\
0, & x < c.
\end{cases} $$
This function is related to our maximization problem via:

Lemma. Let $f \in \overline{X}$, and choose $c \in [0, 1]$ so that $\int_{c}^{1} x \, \mathrm{d}x = \int_{0}^{1} f(x) \, \mathrm{d}x$. Then
$$ \mathsf{Var}(f) \leq \mathsf{Var}(g_c) $$
with equality if and only if $f = g_c$ almost everywhere.
Proof. Since $\int_{0}^{1} g_c(x) \, \mathrm{d}x = \int_{0}^{1} f(x) \, \mathrm{d}x$, we get
\begin{align*}
\mathsf{Var}(g_c) - \mathsf{Var}(f)
&= \int_{0}^{1} (g_c(x)^2 - f(x)^2) \, \mathrm{d}x \\
&= \int_{c}^{1} (x + f(x))(x - f(x)) \, \mathrm{d}x - \int_{0}^{c} f(x)^2 \, \mathrm{d}x \\
&\geq \int_{c}^{1} c(x - f(x)) \, \mathrm{d}x - \int_{0}^{c} c f(x) \, \mathrm{d}x \\
&= c \biggl( \int_{c}^{1} x \, \mathrm{d}x - \int_{0}^{1} f(x) \, \mathrm{d}x \biggr) \\
&= 0.
\end{align*}
Here, we used the inequalities $x + f(x) \geq x \geq c$ for $x \geq c$ and $f(x) \leq x \leq c$ for $x \leq c$. Moreover, this tells that the equality holds if and only if $x - f(x) = 0$ for a.e. $x \in [c, 1]$ and $f(x) = 0$ for a.e. $x \in [0, c]$, proving the equality condition. $\square$

Using this, we are ready to answer OP's question. Let $\alpha = \frac{\sqrt{5} - 1}{2}$. Then it is easy to check that $c \mapsto \mathsf{Var}(g_c)$ is maximized at $c = \alpha$. Now, since $g_{\alpha}$ can be approximated in $L^2$ by the elements of $X$,
$$ \sup_{f \in X} \, \mathsf{Var}(f) = \mathsf{Var}(g_{\alpha}) = \frac{5}{24}(3 - \sqrt{5}) \approx 0.159153. $$
However, this supremum cannot be attained on $X$, because $f \in \overline{X}$ and $\mathsf{Var}(f) = \mathsf{Var}(g_{\alpha})$ together imply that $f = g_{\alpha} \notin X$.
