I've been trying to find the square mean of a gaussian function using the limits of $+/-$ infinity.
$$\int_{-\infty}^{\infty} x^2e^{-2x^2}\mathrm{d}x$$
Why does splitting the function into a $$ u = x$$ and $$v'=xe^{-2x^2}$$ and integrating by parts give a different answer to $$u = x^2$$ and $$v' = e^{-2x^2}$$
If you integrate by parts it doesn't matter in which functions you split your original integral. Both ways will resolve to the same solution but one way might be easier to evaluate. Lets look at both of your proposals and compare the result: $$ \begin{align} \int \underbrace{x}_u\cdot \underbrace{xe^{-2x^2}}_{v'}\text{d}x &= \underbrace{x}_u\cdot\underbrace{\left(-\frac{1}{4}e^{-2x^2}\right)}_v-\int \underbrace{1}_{u'}\cdot\underbrace{\left(-\frac{1}{4}e^{-2x^2}\right)}_v\text{d}x \\ &= -x\frac{1}{4}e^{-2x^2} + \frac{1}{8}\sqrt{\frac{\pi}{2}}\operatorname{erf}\left(\sqrt{2}x\right) \\ &= \frac{1}{8}\sqrt{\frac{\pi}{2}}\operatorname{erf}\left(\sqrt{2}x\right)-\frac{1}{4}e^{-2x^2}x \\ \int \underbrace{x^2}_{u_1}\cdot\underbrace{e^{-2x^2}}_{v_1'}\text{d}x &= \underbrace{x^2}_{u_1}\cdot\underbrace{\frac{1}{2}\sqrt{\frac{\pi}{2}}\operatorname{erf}\left(\sqrt{2}x\right)}_{v_1} - \int \underbrace{2x}_{u_1',u_2}\cdot\underbrace{\frac{1}{2}\sqrt{\frac{\pi}{2}}\operatorname{erf}\left(\sqrt{2}x\right)}_{v_1,v_2'}\text{d}x \\ &= x^2\frac{1}{2}\sqrt{\frac{\pi}{2}}\operatorname{erf}\left(\sqrt{2}x\right) - \left(\underbrace{2x}_{u_2} \cdot \underbrace{\frac{1}{4}\left(\sqrt{2\pi}x\operatorname{erf}\left(\sqrt{2}x\right)+e^{-2x^2}\right)}_{v_2} - \int \underbrace{2}_{u_2'}\cdot\underbrace{\frac{1}{4}\left(\sqrt{2\pi}x\operatorname{erf}\left(\sqrt{2}x\right)+e^{-2x^2}\right)}_{v_2}\text{d}x\right)\\ &\;\; \vdots \\ &= x^2\frac{1}{2}\sqrt{\frac{\pi}{2}}\operatorname{erf}\left(\sqrt{2}x\right)-\frac{1}{16}\left(\sqrt{2\pi}\left(4x^2-1\right)\operatorname{erf}\left(\sqrt{2}x\right)+4e^{-2x^2}x\right) \\ &= \frac{1}{8}\sqrt{\frac{\pi}{2}}\operatorname{erf}\left(\sqrt{2}x\right)-\frac{1}{4}e^{-2x^2}x \end{align} $$
If you split the integrand as $x^2 e^{-2x^2}$ and wish to use integration by parts, you'll have to write $e^{-2x^2}$ as $(\text{erf}(x))^{\prime}$, which is valid but leads to an even worse expression when you do integration by parts.
It seems that there's some misunderstanding.
When you do integration by parts $$\int_a^b uv'dx=uv-\int_a^b u'vdx,$$ you don't perform definite integration to evaluate $v=\int_a^b v'dx.$ You do indifinite integration $v=\int v'dx.$
That makes using $u=x^2$ and $v'=e^{-2x^2}$ not useful because to integrate $v'$ you will get the non-elementary error function.
So, making the fair assumption you probably didn't integrate this monstrosity, I think you integrated definitely which is a common mistake that I do sometimes myself.
$$ \begin{aligned} \int_{-\infty}^{\infty} x^2 e^{-2 x^2} d x = & 2 \int_0^{\infty} x^2 e^{-2 x^2} d x \quad \textrm{ (Even integrand)} \\=&-\frac{1}{2} \int_0^{\infty} x d\left(e^{-2 x^2}\right) \\ = & -\left[\frac{1}{2} x e^{-2 x^2}\right]_0^{\infty}+\frac{1}{2} \int_0^{\infty} e^{-2 x^2} d x \quad \textrm{( By Integration by parts)} \\ = & \frac{1}{2} \cdot \frac{1}{2} \sqrt{\frac{\pi}{2}} \\ = & \frac{1}{4} \sqrt{\frac{\pi}{2}} \end{aligned} $$
