Solving $\int_0^{\infty} \sin(ax^2)\sin(b/x^2)\,\mathrm dx$ The integral is:
$$
\int_{0}^{\infty}\sin\left(ax^{2}\right)\sin\left(\frac b {x^{2}}\right)\,\mathrm{d}x
$$
I already know the answer but I cant prove it! ( You can get the answer using Maple ). I think it should be simplified to a couple of Gaussian integrals ( $\operatorname{erf}$ functions ) but it's not so obvious.
 A: Answer revamped. Here is an answer which is free of heavy computation (with the exception where we utilized the Glasser's master theorem).
If you do not bother with the rigor, it is OK to skip the text and focus on computations.

Let us consider the function
$$ I(a, b) = \int_{0}^{\infty} \exp\left\{ i \left( a x^{2} + \frac{b}{x^2} \right) \right\} \, dx. \tag{*} $$
Denote by $\mathbb{H} = \{z \in \mathbb{C} : \operatorname{Im}(z) > 0 \}$ the upper half-plane and treat $a$ and $b$ as variable on the the closed upper-half plane $\bar{\mathbb{H}}$. Then $I$ defines a holomorphic function on $\mathbb{H}\times\mathbb{H}$. Moreover, a form of Abelian theorem tells that for any $a, b \in \mathbb{R}\setminus\{0\}$ we have
$$ I(a, b) = \lim_{h_1 \to 0^+} \lim_{h_2 \to 0^+} I(a+ih_1, b + ih_2). \tag{1} $$
(The order of the limit above is not so important. So let us just stick to this particular order.) In view of these properties, together with the principle of analytic continuation, it suffices to identify $I(a, b)$ when $a, b$ lie on the positive imaginary axis. So let $a, b > 0$. Then by the Glasser's master theorem,
\begin{align*}
I(ia, ib)
&= \int_{0}^{\infty} \exp\bigg\{ -\left( ax^2 + \frac{b}{x^2} \right) \bigg\} \, dx \\
&= \frac{1}{2}e^{-2\sqrt{ab}} \int_{-\infty}^{\infty} \exp\bigg\{ - a \bigg( x - \frac{\sqrt{b/a}}{x} \bigg)^2 \bigg\} \, dx \\
&= \frac{1}{2}e^{-2\sqrt{ab}} \int_{-\infty}^{\infty} e^{-ax^2} \, dx \qquad \text{($\because$ Glasser)} \\
&= \frac{\sqrt{\pi}}{2\sqrt{a}} e^{-2\sqrt{ab}}
 = \frac{\sqrt{i\pi}}{2\sqrt{ia}} e^{2i\sqrt{ia}\sqrt{ib}}. \tag{2}
\end{align*}
Since the principal square root $z \mapsto \sqrt{z}$ defines a holomorphic function on $\mathbb{H}$, the above computation extends to all of $\mathbb{H}\times\mathbb{H}$, yielding
$$ I(a, b) \stackrel{(2)}{=} \frac{\sqrt{i\pi}}{2\sqrt{a}} e^{2i\sqrt{a}\sqrt{b}}. \tag{3} $$
Then $\text{(1)}$ tells that the result above extends to all of $a, b \in \mathbb{R}\setminus\{0\}$, where the branch for the principal logarithm is chosen to be $-\pi < \arg(z) \leq \pi$. Then applying the relation
$$
\begin{split}
\sin A \sin B
& = \frac{e^{iA} - e^{-A}}{2i} \times \frac{e^{iB} - e^{-iB}}{2i} \\
&= \frac{1}{4}(e^{i(A-B)} + e^{i(B-A)} - e^{i(A+B)} - e^{-i(A+B)}),
\end{split}
\tag{4}
$$
to the identity $\text{(3)}$ and using the relation $\sqrt{-x} = i\sqrt{x}$ for $x > 0$, we obtain
\begin{align*}
\int_{0}^{\infty} \sin(ax^2)\sin\left(\frac{b}{x^2}\right) \, dx
&\stackrel{(4)}{=} \frac{1}{4}(I(a,-b) + I(-a,b) - I(a,b) - I(-a,-b)) \\
&\stackrel{(3)}{=} \frac{(1+i)\sqrt{\pi}}{8\sqrt{2a}}\left( e^{-2\sqrt{ab}} -
 i e^{-2\sqrt{ab}} - e^{2i\sqrt{ab}} + i e^{-2i\sqrt{ab}} \right) \\
&= \frac{\sqrt{\pi}}{4\sqrt{2a}}\left( e^{-2\sqrt{ab}} - \cos(2\sqrt{ab}) + \sin(2\sqrt{ab}) \right).
\end{align*}
A: Consider
$$\int_0^{\infty} dx \, e^{i (a x^2 - b/x^2)}$$
By considering a wedge in the complex plane of angle $\pi/4$ and using Cauchy's theorem, you can show that this integral is equal to
$$e^{i \pi/4} e^{2 \sqrt{a b}} \int_0^{\infty} dx \, e^{-(\sqrt{a} x + \sqrt{b}/x)^2}$$
Let $u = \sqrt{a} x + \sqrt{b}/x$, then
$$dx = \frac{1}{2 \sqrt{a}}\left (1 \pm \frac{u}{\sqrt{u^2-4 \sqrt{a b}}} \right )du$$ 
Note there are two branches, so that integration over $u$ results in two integrals:
$$e^{i \pi/4} \frac{e^{2 \sqrt{a b}}}{2 \sqrt{a}} \left [\int_{2 \sqrt{\sqrt{a b}}}^{\infty} du \left (1 + \frac{u}{\sqrt{u^2-4 \sqrt{a b}}} \right ) e^{-u^2} - \int_{2 \sqrt{\sqrt{a b}}}^{\infty} du \left (1 - \frac{u}{\sqrt{u^2-4 \sqrt{a b}}} \right ) e^{-u^2}\right ] $$
which simplifies to
$$e^{i \pi/4} \frac{e^{2 \sqrt{a b}}}{\sqrt{a}}\int_{2 \sqrt{\sqrt{a b}}}^{\infty} du  \frac{u}{\sqrt{u^2-4 \sqrt{a b}}} e^{-u^2}$$
which turns out to be very integrable; the result is
$$\int_0^{\infty} dx \, e^{i (a x^2 + b/x^2)}=e^{i \pi/4} \sqrt{\frac{\pi}{a}}e^{-2 \sqrt{a b}}$$
Now consider
$$\int_0^{\infty} dx \, e^{i (a x^2 + b/x^2)}$$
By the same reasoning, this integral is equal to
$$e^{i \pi/4} \sqrt{\frac{\pi}{a}}e^{-i 2 \sqrt{a b}}$$
The desired integral is one-half the real part of the difference between these two integrals, i.e.,
$$\int_0^{\infty} dx \sin{\left ( a x^2\right )} \sin{\left (\frac{b}{x^2}\right)} = \frac12 \Re{\left[\int_0^{\infty} dx \, e^{i (a x^2 - b/x^2)} - \int_0^{\infty} dx \, e^{i (a x^2 + b/x^2)} \right]}$$
Therefore, converting the exponential to hyperbolic functions (for symmetry), I get
$$\int_0^{\infty} dx \sin{\left ( a x^2\right )} \sin{\left (\frac{b}{x^2}\right)} = \frac14 \sqrt{\frac{\pi}{2 a}} \left[\left (\cosh{\left(2 \sqrt{a b}\right)}-\cos{\left(2 \sqrt{a b}\right)}\right) - \left (\sinh{\left(2 \sqrt{a b}\right)}-\sin{\left(2 \sqrt{a b}\right)}\right) \right]$$
A: Notice that the function 
$$f(x)=e^{2\alpha\beta}\mathrm{erf}\left(\alpha x+\frac{\beta}{x}\right)+e^{-2\alpha\beta}\mathrm{erf}\left(\alpha x-\frac{\beta}{x}\right)$$
has the derivative
$$f'(x)=\frac{4\alpha}{\sqrt{\pi}}\,e^{-\alpha^2 x^2-\beta^2/x^2}.$$
This allows to express the antiderivative $\displaystyle \int \sin ax^2\sin\frac{b}{x^2}dx$ as a linear combination of eight $\mathrm{erf}$-functions. To compute the corresponding definite integral, it may be helpful to consider $x$ as complex variable and turn the contour of integration by $\frac{\pi}{4}$ (the direction of this turn may be different for different exponential pieces; also, one may need to decompose the interval $(0,\infty)$ into two pieces $(0,1]\cup[1,\infty)$).
A: $$\int_0^{\infty} \sin ax^2\sin bx^{-2}\,dx=\frac{1}{2}\int_0^{\infty} \cos \left(ax^2-\frac{b}{x^2}\right)\,dx-\frac{1}{2}\int_0^{\infty} \cos \left(ax^2+\frac{b}{x^2}\right)\,dx$$
In general:
$$\begin{aligned} \int_{0}^{\infty}f \left(p x-\frac{q}{x}\right)^2\,dx &= \sqrt{\frac{q}{p}}\int_{0}^{\infty  }f \left(\sqrt{pq} x- \frac{\sqrt{pq}}{x}\right)^2\,dx \quad(\textstyle x\to  \sqrt{\frac{q}{p}}x )\\ &=\sqrt{\frac{q}{p}}\left[ \int_{0}^{1} f \left(\sqrt{pq} x-\frac{\sqrt{pq}}{x} \right)^2\,dx+ \int_{1}^{\infty} f \left(\sqrt{pq}x-\frac{\sqrt{pq}}{x}\right)^2\,dx \right]\\&= \sqrt{\frac{q}{p}}\left[\int_{1}^{\infty} \left(1+\frac{1}{x^2}\right)f \left(\sqrt{pq} x-\frac{\sqrt{pq}}{x} \right)^2\,dx \right]\quad( x\to x^{-1})\\&=\frac{1}{p} \int_{0}^{\infty} f (t^2 )\,dt \quad \big(t=\sqrt{pq} x-\sqrt{pq}x^{-1}\big) \end{aligned}$$
Observe that:
$$\begin{aligned}\int_0^{\infty} \cos \left(ax^2+\frac{b}{x^2}\right)\,dx&=\cos 2\sqrt{ab}\int_{0}^{\infty} \cos \left(\sqrt{a}x-\frac{\sqrt{b}}{x}\right)^2\,dx\\&\;-\sin 2\sqrt{ab}\int_{0}^{\infty} \sin \left(\sqrt{a}x-\frac{\sqrt{b}}{x}\right)^2\,dx\end{aligned}$$
Now since $\displaystyle \int_0^{\infty}\cos x^2\,dx=\int_0^{\infty}\sin x^2\,dx=\sqrt{\frac{\pi}{8}}$ using our previous result we have:
$$\int_0^{\infty} \cos \left(ax^2+\frac{b}{x^2}\right)\,dx=\sqrt{\frac{\pi}{8a}}\left(\cos 2\sqrt{ab}-\sin 2\sqrt{ab}\right)$$
Similarly, writing:
$$\begin{aligned}\int_0^{\infty} \cos \left(ax^2-\frac{b}{x^2}\right)\,dx&=\cosh 2\sqrt{ab}\int_{0}^{\infty} \cos \left(\sqrt{a}x-i\frac{\sqrt{b}}{x}\right)^2\,dx\\&\;-\sinh 2\sqrt{ab}\int_{0}^{\infty} \sin \left(\sqrt{a}x-i\frac{\sqrt{b}}{x}\right)^2\,dx\end{aligned}$$
We get:
$$\int_0^{\infty} \cos \left(ax^2-\frac{b}{x^2}\right)\,dx=\sqrt{\frac{\pi}{8a}}\left(\cosh 2\sqrt{ab}-\sinh 2\sqrt{ab}\right)$$
Putting everything together:
$$\int_0^{\infty} \sin ax^2\sin bx^{-2}\,dx=\frac{1}{2}\sqrt{\frac{\pi}{8a}}\Bigg\{\big(\cosh 2\sqrt{ab}-\sinh 2\sqrt{ab}\big)-\big(\cos 2\sqrt{ab}-\sin 2\sqrt{ab}\big)\Bigg\}$$
