It seems to me that requesting the smallest possible value for $a^2+b^2$ is almost a red herring.
Claim.
For $|a| \leq 4$ and $b > 2 |a| - 2$, the polynomial
$$
p(x) = x^4 + ax^3 + bx^2 + ax + 1
$$
has no real zeros.
Proof.
First set $b = 2|a| - 2 + \epsilon$, where $\epsilon > 0$. If we denote $\pm = \operatorname{sign} a$, so that $a = \pm |a|$, we have
$$
\begin{align}
p(x) &= x^4 + ax^3 + (2|a|-2+\epsilon)x^2 + ax + 1 \\
&= x^4 - 2x^2 + 1 + x(ax^2 + 2|a|x + a) + \epsilon x^2 \\
&= (x^2-1)^2 + ax(x \pm 1)^2 + \epsilon x^2 \\
&= (x \pm 1)^2\left[ (x \mp 1)^2 + ax \right] + \epsilon x^2 \\
&= (x \pm 1)^2\left[ x^2 \pm (|a|-2)x + 1 \right] + \epsilon x^2 \\
&\geq (x \pm 1)^2 \min\left\{x^2 - 2x + 1,x^2+2x+1\right\} + \epsilon x^2 \\
&= (x \pm 1)^2\min\left\{(x-1)^2,(x+1)^2\right\} + \epsilon x^2 \\
&> 0.
\end{align}
$$
Q.E.D.
The points on the boundary of the region
$$
A = \{(a,b) \,\colon |a| \leq 4 \,\,\,\text{and}\,\,\,b > 2 |a| - 2\}
$$
which have the smallest norm are
$$
(a,b) = \left(\pm \frac{4}{5},-\frac{2}{5}\right),
$$
as shown in the following image.

It only remains to demonstrate that one of these points yields a polynomial with a real zero. In fact they both do:
$$
x^4 + \frac{4}{5} x^3 - \frac{2}{5} x^2 + \frac{4}{5} x + 1
$$
has a zero at $x=-1$ and
$$
x^4 - \frac{4}{5} x^3 - \frac{2}{5} x^2 - \frac{4}{5} x + 1
$$
has a zero at $x=1$.
Thus
The smallest value of $a^2 + b^2$ for which we can find some $a,b\in \mathbb R$ such that $p(x)$ has a real zero is
$$
\left(\frac{4}{5}\right)^2 + \left(\frac{2}{5}\right)^2 = \frac{4}{5}.
$$
Below is a plot of the $(a,b)$-plane which shows the region (in blue) where the polynomial $p(x)$ has at least one real root. Note that for $|a| \leq 4$ the boundary of this region is precisely $b = 2|a| - 2$ (shown in black), but for $|a| > 4$ it curves inward.
