The global maximum of a continuous and differentiable real-valued function $f(x)$ on some interval $[a,b]$ occurs either at one of the endpoints, or at a critical point $x = x_0$ satisfying $f'(x_0) = 0$. Thus, by computing the derivative and finding the set of critical points, you obtain a set of candidate extrema, after which you can use other arguments to find which one(s) correspond to a global maximum of $f$.
So in this case, you know that you are restricted to $[a,b] = [0,1]$. You also know that $f(x) \ge 0$ on this interval. You can compute fairly trivially that $f(0) = f(1) = 0$. After finding that $x_0 = 1/\sqrt{3}$ gives $f'(x_0) = f'(1/\sqrt{3}) = 0$, and $f(1/\sqrt{3}) > 0$, you know that this is a critical point and it is better than the endpoints, so the global maximum is not at either endpoint. And since it is the only critical point on the given interval, it certainly isn't a minimum, so it's a maximum.
To be even more sure, you can calculate the second derivative $f''(x) = -24x$, and you can conclude that on $x \in [0,1]$, this function is always nonpositive, thus the function is concave down. Thus the mode is $X = 1/\sqrt{3}$ as claimed. The probability that a single observation is less than the mode is $$\Pr[X < 1/\sqrt{3}] = \int_{x=0}^{1/\sqrt{3}} f_X(x) \, dx = \int_{x=0}^{1/\sqrt{3}} 4(x-x^3) \, dx.$$ The probability that three independent observations are all each less than the mode is simply $$\Pr\left[\bigcap_{i=1}^3 X_i < \frac{1}{\sqrt{3}} \right] \overset{\text{iid}}{=} \Pr[X < 1/\sqrt{3}]^3,$$ that is to say, it is simply the probability of a single observation being less than the mode, raised to the third power.