At first glance $(\max(0, x))^2$ looks smooth.
If I'm not mistaken:
if $x \le 0$ the derivative equals $2 \times\max(0, x) \times0 = 0$
if $x \ge 0$ the derivative equals $2 \times \max(0, x) \times 1$ and it equal zero if $x = 0$
Looks like derivative of $(\max(0, x))^2$ is continuous.
But when I tried to check with Wolfram Alpha it shows me that derivative of $(\max(0, x))^2$ is indeterminate at $x = 0$. Link
So can anyone tell me where I made a mistake?
Your function is obviously differentiable on $\mathbb R^*$.
You have if you call your function $f$:
$$\forall x<0, \quad f'(x)=0$$
and
$$\forall x>0, \quad f'(x)=2x.$$
So
$$\lim_{\substack{ x\to 0 \\ x<0}} f'(x)=0=\lim_{\substack{ x\to 0 \\ x>0}} f'(x).$$
And because $f$ is continuous on $\mathbb R$, it gives you that $f$ is differentiable on $0$.
(Wolfram Alpha is wrong)
Let $f(x) = {\left[\max(0, x)\right]}^2$. The derivative is indeed $f'(x) = \max(0, 2x)$ and the directional limits at $0$ agree.
However, smooth usually means $\mathcal C^\infty$, but $f$ is not $\mathcal C^\infty$, because $f''$ is not continuous. We have that $$f''(x) = 2H(x),$$ where $H(x)$ is the Heaviside step function.
EDIT: The question originally asked if the function is smooth, and was later changed.
Another way to write the function is: $$\begin{align} f(x) & = \operatorname{max}(0, x)^2\\ & = x^2 H(x),\end{align}$$ with $H$ the Heaviside step function. Taking the derivative of this second form gives: $$ f'(x) = 2x H(x) + x^2 \delta(x). $$ If you naively insert $x=0$ into this expression you'll get $f' = 0 + 0 \times \infty$, a classic indeterminate form. This is cleared up by recognizing that in an integral, the only place a delta function has unambiguous meaning, the delta function evaluates what it multiplies at the point where its argument is $0$, divided by the derivative of the argument. In this case, that would give $f'(0) = 0$.
