$$\int \frac{2x^3+3x^2}{(2x^2+x-3)\sqrt{x^2+2x-3}}\mathrm dx$$$$=2\int \frac{x^3}{(2x^2+x-3)\sqrt{x^2+2x-3}}\mathrm dx+3\int \frac{x^2}{(2x^2+x-3)\sqrt{x^2+2x-3}}\mathrm dx$$
For these two integrals, I have tried Euler substitutions and various types of substitutions of irrational functions, but it seems that those substitutions don't simplify the integral.
What substitution is useful for this type of integrals?
Since $x^2 + 2x -3 = (x+1)^2 -4$, you can use the substitution $x+1 = 2 \cosh t$. This way you are going to get a rational function in $\cosh t$ and $\sinh t$.
$$\int\frac{2x^3+3x^2}{\left(2x^2+x-3\right)\sqrt{x^2+2x-3}}\space\text{d}x=$$
Using the Euler subsitution:
$x=\frac{u^2+3}{2u+2}$ and $\text{d}x=\left(\frac{2u}{2u+2}-\frac{2\left(u^2+3\right)}{\left(2u+2\right)^2}\right)\space\text{d}u$.
And when we substitute back we get $u=x+\sqrt{x^2+2x-3}$:
$$\int\frac{\left(u^2+3\right)^2}{2\left(u^2-1\right)^2}\space\text{d}u=\frac{1}{2}\int\frac{\left(u^2+3\right)^2}{\left(u^2-1\right)^2}\space\text{d}u=$$ $$\frac{1}{2}\int\left[\frac{4}{(u+1)^2}+\frac{4}{(u-1)^2}+1\right]\space\text{d}u=$$ $$\frac{1}{2}\left[4\int\frac{1}{(u+1)^2}\space\text{d}u+4\int\frac{1}{(u-1)^2}\space\text{d}u+\int1\space\text{d}u\right]=$$ $$\frac{1}{2}\left[4\int\frac{1}{(u+1)^2}\space\text{d}u+4\int\frac{1}{(u-1)^2}\space\text{d}u+u\right]=$$
Substitute $s=u+1$ and $\text{d}s=\text{d}u$.
And Substitute $p=u-1$ and $\text{d}p=\text{d}u$:
$$\frac{1}{2}\left[4\int\frac{1}{s^2}\space\text{d}s+4\int\frac{1}{p^2}\space\text{d}p+u\right]=$$ $$\frac{1}{2}\left[u-\frac{4}{s}-\frac{4}{p}\right]+\text{C}=$$ $$\frac{1}{2}\left[u-\frac{4}{u+1}-\frac{4}{u-1}\right]+\text{C}=$$ $$\frac{1}{2}\left[x+\sqrt{x^2+2x-3}-\frac{4}{x+\sqrt{x^2+2x-3}+1}-\frac{4}{x+\sqrt{x^2+2x-3}-1}\right]+\text{C}=$$ $$\frac{1}{2}\left[\frac{(3+x)(2x-3)}{\sqrt{(3+x)(x-1)}}\right]+\text{C}=\frac{(3+x)(2x-3)}{2\sqrt{(3+x)(x-1)}}+\text{C}$$
Hint:
The numerator of the original integral can be written
$$2x^3 + 3x^2 = (2x+3)(x^2)$$
The denominator of the original integral can be written
$$\left(2x^2 + x - 3\right)\sqrt{x^2 + 2x - 3} = (2x+3)(x-1)\sqrt{(x+3)(x-1)}$$
The $(2x+3)$ terms cancel, so the integral becomes
$$\displaystyle\int \frac{x^2}{(x-1)^{3/2}(x+3)^{1/2}}\,\mathrm{d}x$$
Hint:
Let $$I=\int \frac{2x^3+3x^2}{(2x^2+x-3)\sqrt{x^2+2x-3}}\mathrm dx$$ Then, we have \begin{align*} I&=\int \frac{2x^3+x^2-3x+2x^2+x-3+2x+3}{(2x^2+x-3)\sqrt{x^2+2x-3}}\mathrm dx\\ &=\int\frac{x+1}{\sqrt{x^2+2x-3}}\mathrm d x+\int\frac{2x+3}{(2x^2+x-3)\sqrt{x^2+2x-3}}\mathrm dx\\ &=\int\frac{x+1}{\sqrt{x^2+2x-3}}\mathrm d x+\int\frac{2x+3}{(2x+3)(x-1)\sqrt{x^2+2x-3}}\mathrm dx\\ &=\int\frac{x+1}{\sqrt{x^2+2x-3}}\mathrm d x+\int\frac{1}{(x-1)\sqrt{x^2+2x-3}}\mathrm dx\\ \end{align*} Now, we can make $x+1=2\sec t$
