Here is a laborious proof by contradiction.
Suppose $|f''(x)| \leq 4$ for all $x$.
Define $\phi(x) = f(x)-x$. Clearly $\phi'(x) = f'(x)-1$, $\phi''(x) = f''(x)$.
Furthermore, $\phi(0) = \phi(1) = 0$, $\phi'(0) = \phi'(1) = -1$.
The MVT gives $\phi(x) = \phi(0)+\phi'(0) x + \frac{1}{2}\phi''(c_x) x^2$ for some $c_x \in [0,x]$, which yields the estimate (using $|\phi''(x)| \leq 4$) $\phi(x) \leq -x + 2 x^2$. In particular, $\phi(x) <0$ for $x\in (0,\frac{1}{2})$. A similar analysis, expanded around $x=1$ shows that $\phi(x) >0$ for $x\in (\frac{1}{2}, 1)$. It follows that $\phi(\frac{1}{2}) = 0$.
Now let $\eta(x) = -x + 2 x^2-\phi(x)$. From above, we have $\eta(x)\geq 0$ for $x\in [0,\frac{1}{2}]$. We will show that, in fact, $\eta(x) = 0$. We have $\eta(0) = \eta(\frac{1}{2}) = 0$, hence $\eta$ is maximized at some $\hat{x} \in (0,\frac{1}{2})$. Then using the MVT, we have $\eta(\frac{1}{2}) = \eta(\hat{x}) + \eta'(\hat{x}) (\frac{1}{2}-\hat{x}) + \frac{1}{2}\eta''(c) (\frac{1}{2}-\hat{x})^2$. Since $\eta'(\hat{x})=0$ and $\eta''(c) \geq 0$, this implies that $\eta(\hat{x}) \leq \eta(\frac{1}{2}) = 0$. Hence $\eta(x) = 0$ for $x \in [0,\frac{1}{2}]$ and so $\phi(x) = -x+2x^2$ on this interval.
A similar analysis for the interval $[\frac{1}{2}, 1]$ shows that $\phi(x) = -2x^2+3x-1$ on this interval.
This gives $\phi'(x) = \begin{cases}4x-1, & x \in [0,\frac{1}{2}] \\ 3-4x, & x \in [\frac{1}{2}, 1] \end{cases}$, or $\phi'(x) = 1 -4 | x-\frac{1}{2}|$. Since $\phi'$ is not differentiable at $x=\frac{1}{2}$, $f'$ is not differentiable at $x=\frac{1}{2}$, which is a contradiction.