Basically we are trying to find the volume of the sphere $x^2+y^2+z^2 \leq 1$ between two cones $z^2 = x^2 + y^2$ and $z^2 = 3(x^2+y^2)$.
So the bounds are much easier to set up in spherical coordinates.
$x = \rho \cos \theta \sin \phi, y = \rho \sin \theta \sin \phi, z = \rho \cos \phi$
$0 \leq \rho \leq 1, 0 \leq \theta \leq \frac{\pi}{2}$ are obvious as we are in first octant and radius of the sphere is $1$.
Now to find the bounds of $\phi$, we observe that
$x^2 + y^2 \leq z^2 \leq 3(x^2+y^2)$
plugging in $x,y,z$, $ \frac{1}{\sqrt3} \leq \tan \phi \leq 1 \implies \frac{\pi}{6} \leq \phi \leq \frac{\pi}{4}$
$xyz = \rho^3 \cos\theta \sin \theta \sin^2\phi \cos\phi$
So the integral becomes,
$\displaystyle \int_{0}^{\pi/2} \int_{\pi/6}^{\pi/4} \int_0^1 \rho^5 \cos\theta \sin \theta \sin^3\phi \cos\phi \ d\rho \ d\phi \ d\theta$
I did not do the integral by hand but WolframAlpha shows the result as $\displaystyle \frac{1}{256}$.