Integrating $\frac{x^3}{\sqrt{x^2 + 4x + 6}}$ Question:
Evaluate $\displaystyle\int \frac{x^3}{\sqrt{x^2 + 4x + 6}}\ dx $.
My attempt:
$\begin{align} \int \frac{x^3}{\sqrt{x^2 + 4x + 6}}\ dx & = \int \frac{x^3}{\sqrt{(x+2)^2 + 2}}\ dx \\& \overset{(1)}= \int \frac{(\sqrt{2} \tan(t) - 2 )^3 \sqrt{2} \sec^2(t)\ }{\sqrt{2\tan^2(t) + 2}}\ dt\\& = \frac{1}{\sqrt{2}}\int\frac{(\sqrt{2} \tan(t) - 2 )^3 \sqrt{2} \sec^2(t)}{\sec(t)}\ dt\\& = \int(\sqrt{2} \tan(t) - 2 )^3 \sec(t)\ dt\\& = \int -8 \sec(t) + 2 \sqrt2 \tan^3(t) \sec(t) - 12\tan^2(t) \sec(t)\\&\qquad + 12 \sqrt2 \tan(t) \sec(t)\ dt\\\\& = -8\ln|\sec(t) + \tan(t)| + 12\sqrt{2} \sec{(t)}  \\
&\qquad + 2\sqrt{2} \int \tan^3(t) \sec(t) \ dt - 12\int\tan^2(t) \sec(t) \ dt\end{align}$
Now both of these integrals can be evaluated using some sort of substitution and integration by parts rule.
This would give us,
$$\boxed{-8\ln|\sec(t) + \tan(t)| + 12\sqrt{2} \sec{(t)}  + 2\sqrt{2}\left[\frac{\sec^3(t)}{3} - \sec(t)\right] - 12\left[\frac{-1}{2} \ln|\sec t + \tan t| + \frac12 \sec t \tan t \right] + C}$$


$(1)$ Here I've made a substitution $t = \tan^{-1}\left(\frac{x+ 2}{\sqrt 2}\right)$.


Wolframalpha gives answer as $$\boxed{\frac{1}{3}(x^2 - 5x + 18) \sqrt{x^2 +4x + 6}  - 2 \sinh^{-1}\left(\frac{x+2}{\sqrt{2}}\right) + C}$$
How would I simplify my answer equal to this? Undoing my substitution $t = \tan^{-1}\left(\frac{x+ 2}{\sqrt 2}\right)$ is also not an easy task I think.
 A: For indefinite integral of the form
$$I_n=\int \frac{x^n}{\sqrt{x^2 + bx + c}}\ dx 
$$
it is advised that $I_n$ be reduced first to $I_0$ before any substitution. This is achieved by the reduction formula below with $f(x)=x^2+bx+c$
$$\int \frac{f’(x)^{n}}{\sqrt{f(x)}}dx= K_n = \frac2nf’(x)^{n-1}\sqrt{f(x)}+\frac{n-1}n (b^2-4c)K_{n-2}
$$
Thus, apply it to the integral to obtain
\begin{align}
&\int \frac{x^3}{\sqrt{x^2 + 4x + 6}}\ dx \\
=&\ \frac18\int \frac{(2x+4)^3-12(2x+4)^2+48(2x+4)-64}{\sqrt{x^2 + 4x + 6}}\ dx \\
=&\ \frac13(x^2-5x+18)\sqrt{x^2 + 4x + 6}-2\int \frac{1}{\sqrt{x^2 + 4x + 6}}\ dx 
\end{align}
A: By hyperbolic substitution
Letting $x+2=\sqrt 2 \sinh \theta$ transforms the integral into
$$
\begin{aligned}
I= & \int \frac{(\sqrt{2} \sinh \theta-2)^3}{\sqrt{2} \cosh \theta} \cdot \sqrt{2} \cosh \theta d \theta \\
= & \int\left(2 \sqrt{2} \sinh ^3 \theta-12 \sinh ^2 \theta+12 \sqrt{2} \sinh \theta-8\right) d \theta \\
= & 2 \sqrt{2} \int\left(\cosh ^2 \theta-1\right) d(\cosh \theta)-12 \int \frac{\cosh 2 \theta-1}{2} d \theta +12 \sqrt{2} \cosh \theta-8 \theta \\
= &\frac{2 \sqrt{2}\cosh ^3 \theta}{3}-3\sinh 2\theta +10\sqrt{2} \cosh \theta-2 \theta+C \cdots (*)\\=& \frac{2 \sqrt{2}}{3} \cosh \theta\left(15+\cosh ^2 \theta-\frac{9}{\sqrt{2}} \sinh \theta\right)-2 \theta+C
\end{aligned}
$$
Putting back $x+2=\sqrt 2 \sinh \theta$ yields
$$
\begin{aligned}
I & =\frac{2 \sqrt{2}}{3} \sqrt{1+\left(\frac{x+2}{\sqrt{2}}\right)^2}\left[15+1+\left(\frac{x+2}{\sqrt{2}}\right)^2-\frac{9}{\sqrt{2}} \cdot \frac{x+2}{\sqrt{2}}\right] +C\\
& =\frac{1}{3} \sqrt{x^2+4 x+6}\left(x^2-5 x+18\right)-2 \sinh ^{-1}\left(\frac{x+2}{\sqrt{2}}\right)+C
\end{aligned}
$$
By trigonometric substitution
Questioner using $x+2=\sqrt 2\tan t$ got the answer
$$I=-8\ln|\sec(t) + \tan(t)| + 12\sqrt{2} \sec{(t)}  + 2\sqrt{2}\left[\frac{\sec^3(t)}{3} - \sec(t)\right] - 12\left[\frac{-1}{2} \ln|\sec t + \tan t| + \frac12 \sec t \tan t \right] + C$$
Using $x+2=\sqrt2\tan t =\sqrt 2\sinh \theta \Leftrightarrow \tan t=\sinh \theta \textrm{ and } \sec t=\cosh \theta$, we get
$$
\begin{aligned}
I=-2 & \ln |\sec t+\tan t|+\frac{2 \sqrt{2}}{3} \sec ^3 t+10 \sqrt{2} \sec t -6 \sec t \tan t+C \\
=- & 2 \ln |\cosh \theta+\sinh \theta|+\frac{2 \sqrt{2}}{3} \cosh ^3 \theta+10\sqrt{2} \cosh \theta -6 \cosh \theta \sinh \theta+C
\end{aligned}
$$
Since
$\ln |\cosh \theta+\sinh \theta |=  \ln \left(\frac{e^\theta+e^{-\theta}}{2}+\frac{e^\theta-e^{-\theta}}{2}\right) 
= \theta$, therefore $$I= \frac{2 \sqrt{2}\cosh ^3 \theta}{3}-3\sinh 2\theta +10\sqrt{2} \cosh \theta-2 \theta+C \cdots (*)$$
We can now conclude that both substitutions gives the same result.
A: Undoing the substitution is exactly how you would match the two results.
By the Pythagorean theorem, a right triangle with a reference angle $t$ such that $\tan(t)=\frac{x+2}{\sqrt2}$ has its sides occurring in a ratio of $\sqrt2$ (leg adjacent to $t$) to $x+2$ (leg opposite $t$) to $\sqrt{(x+2)^2+2}=\sqrt{x^2+4x+6}$ (hypotenuse). It follows that
$$t = \tan^{-1}\left(\frac{x+2}{\sqrt2}\right) \\
\implies 
\begin{cases} \tan(t) = \frac{x+2}{\sqrt2} \\ 
\sec(t)=\frac{\sqrt{x^2+4x+6}}{\sqrt2} \\ 
\sec(t)+\tan(t) = \frac{\sqrt{x^2+4x+6}+x+2}{\sqrt2} \\
\sec(t)\tan(t) = \frac{(x+2)\sqrt{x^2+4x+6}}2 \\
\ln\left|\sec(t)+\tan(t)\right| = \ln\left(\frac{\sqrt{x^2+4x+6}+x+2}{\sqrt2}\right)
\end{cases}$$
After making the replacements, your antiderivative reduces to
$$-2 \ln\left(\frac{\sqrt{x^2+4x+6}+x+2}{\sqrt2}\right) + (4-3x) \sqrt{x^2+4x+6} + \frac13 (x^2+4x+6)^{3/2} + C$$
Factorize the last two terms as
$$\bigg(4-3x + \frac13 (x^2+4x+6)\bigg) \sqrt{x^2+4x+6} = -\frac13 (x^2+5x-18) \sqrt{x^2+4x+6}$$
Finally, the logarithm can be rewritten using the definition of the inverse hyp. sine:
$$\begin{align*}
\ln\left(\frac{\sqrt{x^2+4x+6}+x+2}{\sqrt2}\right) &= \ln\left(\sqrt{\frac{(x+2)^2+2}2} + \frac{x+2}{\sqrt2}\right) \\
&= \ln\left(\sqrt{\left(\frac{x+2}{\sqrt2}\right)^2+1} + \frac{x+2}{\sqrt2}\right) \\[1ex]
&= \sinh^{-1}\left(\frac{x+2}{\sqrt2}\right)
\end{align*}$$
