Used a different version of the secant reduction formula. I really enjoyed this problem!
Let $x=\sec(u)$ and $dx=\sec(u)\tan(u)\,dx$ so that:
$$\int x^2\sqrt{x^2-1}\,dx=\int\sec^2(u)\sqrt{\sec^2(u)-1}\sec(u)\tan(u)\,du$$
Now simplify, noting that $\sec^2(u)-1=\tan^2(u)$:
$$=\int\sec^3(u)\tan^2(u)\,du$$
And again, using $\sec^2(u)-1=\tan^2(u)$ and simplifying:
$$\int(\sec^5(u)-\sec^3(u))\,du$$
Apply the version of the secant reduction formula containing tangent, which is:
$$\int\sec^n(u)\,du=\frac{\tan(u)\sec^{n-2}(u)}{n-1}+\frac{n-2}{n-1}\int\sec^{n-2}(u)\,du$$
To get:
$$\int(\sec^5(u)-\sec^3(u))\,du=\frac{\tan(u)\sec^3(u)}{4}+\frac{3}{4}\bigg(\frac{\tan(u)\sec(u)}{2}+\frac{1}{2}\ln|\sec(u)+\tan(u)|\bigg)$$
$$-\bigg(\frac{\tan(u)\sec(u)}{2}+\frac{1}{2}\ln|\sec(u)+\tan(u)|\bigg)$$
Next apply the substitutions $x=\sec^{-1}(u)$ and $\tan(\sec^{-1}(u)=x\sqrt{1-\frac{1}{x^2}}$ and simplify:
$$\int x^2\sqrt{x^2-1}\,dx=\frac{1}{4}x^4\sqrt{1-\frac{1}{x^2}}-\frac{1}{8}x^2\sqrt{1-\frac{1}{x^2}}-\frac{1}{8}\ln\bigg|x+\sqrt{1-\frac{1}{x^2}}\bigg|+C$$