Starting with
$$
m\frac{d^2x}{dt^2}=-kx + \alpha x^3
$$
(a)for small x: $kx > \alpha x^{3}$ then force is retarding when $kx < \alpha x^{3}$ the force accelerates. So in the latter case the mass does not always return to zero.
(b)
Taking the rhs of the above equation as $V(x)$ we find the equilibrium points as the roots of $V(x)$ which occur at
$$
x_{0} = 0\\
x_{0} = \pm\sqrt{\frac{k}{\alpha}}
$$
to find the nature of the stability points we can perturb the solution about $x_{0}$ as $x=x_{0} + \delta x$. inserting into the original equation and linearise worth respect to $\delta x$ we find
$$
\frac{d^{2}\delta x}{dt^{2}} = \frac{dV\left(\delta x\right)}{dx} = \left(-k + 3\alpha x_{0}^{2}\right)\delta x
$$
Then you would solve for $\delta x$ and then plug the fixed points into the resulting equation for $\delta x$ and depending on the nature of $\delta x$'s will determine if the points are stable,unstable etc etc.
$\textbf{Appendix}$
we can use the identity
$$
\frac{d^2x}{dt^2} = p\frac{dp}{dx} = \frac{d}{dx}\left(\frac{p^{2}}{2}\right)
$$
where $p=\frac{dx}{dt}$.
Subbing into the original equation we have
$$
m\frac{d}{dx}\left(\frac{p^{2}}{2}\right) = -kx + \alpha x^3
$$
this yields
$$
\frac{p^{2}}{2} = \frac{1}{m}\left(-k\frac{x^{2}}{2} + \alpha\frac{x^{4}}{4} + C\right)
$$
Now in physics we can either keep C or choose it to be zero. If we focus on C = 0 case
$$
p = x\sqrt{\frac{\alpha}{2m}\left( x^{2} -\frac{2k}{\alpha}\right)} = \frac{dx}{dt}
$$
this leads to
$$
\int \frac{1}{x\sqrt{\left( x^{2} -\frac{2k}{\alpha}\right)}}dx = \int \sqrt{\frac{\alpha}{2m}}dt
$$
transforming again with $u = \frac{x}{\sqrt{\frac{2k}{\alpha}}}$
the lhs becomes
$$
\frac{1}{\sqrt{\frac{2k}{\alpha}}}\int \frac{1}{u\sqrt{u^{2}-1}}du = \sqrt{\frac{\alpha}{2m}}t + C_{1}
$$
making a final substitution of $u = sec(x)$ we can solve the lys integral to become
$$
\frac{1}{\sqrt{\frac{2k}{\alpha}}}u = \sqrt{\frac{\alpha}{2m}}t + C_{1}
$$
with the final solution of
$$
x(t) = \mathrm{sec}^{-1}\left(\sqrt{\frac{k}{m}}t + C_{1}\right)
$$