Evaluate : $\int _0^{\infty } \sin (ax^2)\cos \left(2bx\right)dx$ How can I solve 
$$\int_{0}^{\infty} \sin  \left (ax^2 \right) \cos \left (2bx \right)dx$$
Thanks!
 A: I will assume that $a \gt 0$.  One way to attack this integral is to extend the integration region to the whole real line by symmetry, and then split up the product into a sum of sines.  Thus, the integral is
$$\frac14 \int_{-\infty}^{\infty} dx \, \sin{\left ( a x^2+2 b x \right )} +  \frac14 \int_{-\infty}^{\infty} dx \, \sin{\left ( a x^2-2 b x \right )}$$ 
which may be expressed as
$$\frac14 \operatorname{Im}{\left [\int_{-\infty}^{\infty} dx \, e^{i (a x^2+2 b x)} \right ]}+\frac14 \operatorname{Im}{\left [\int_{-\infty}^{\infty} dx \, e^{i (a x^2-2 b x)} \right ]} $$
Complete the squares and get
$$\frac14 \operatorname{Im}{\left [ e^{-i b^2/a} \int_{-\infty}^{\infty} dx \, e^{i a (x+b/a)^2} \right ]} + \frac14 \operatorname{Im}{\left [ e^{-i b^2/a} \int_{-\infty}^{\infty} dx \, e^{i a (x-b/a)^2} \right ]} $$
Note that both integrals are equal - we just shift the respective origins.  So we now have the integral being equal to
$$\frac12 \operatorname{Im}{\left [ e^{-i b^2/a} \int_{-\infty}^{\infty} dx \, e^{i a x^2} \right ]}$$
The integral converges and may be shown to be equal to $\sqrt{\pi/(-i a)}$.  Thus, we find that

$$\int_0^{\infty} dx \, \sin{a x^2} \, \cos{2 b x} = \frac12 \sqrt{\frac{\pi}{2 a}} \left (\cos{\frac{b^2}{a}}-\sin{\frac{b^2}{a}} \right ) $$

A: Concerning the antiderivative, start writing$$\sin  \left (ax^2 \right) \cos \left (2bx \right)=\frac12\left(\sin(ax^2+2bx)+\sin(ax^2-2bx)\right)$$ Now $$ax^2+2bx=\left(\sqrt a x +\frac{b}{\sqrt a}\right)^2-\frac{b^2}a$$ $$ax^2-2bx=\left(\sqrt a x -\frac{b}{\sqrt a}\right)^2-\frac{b^2}a$$ Now, consider $$I=\int \sin(ax^2+2bx)\,dx$$ and change variable accordingly for each sine. Expand the sine and you will arrive to a linear comination of sine and cosine Fresnel integrals. Back to $x$, you should get $$I=\frac{\sqrt{\frac{\pi }{2}} \left(\cos \left(\frac{b^2}{4 a}\right) S\left(\frac{b+2
   a x}{\sqrt{a} \sqrt{2 \pi }}\right)-\sin \left(\frac{b^2}{4 a}\right)
   C\left(\frac{b+2 a x}{\sqrt{a} \sqrt{2 \pi }}\right)\right)}{\sqrt{a}}$$ Similarly  $$J=\int \sin(ax^2-2bx)\,dx$$ $$J=-\frac{\sqrt{\frac{\pi }{2}} \left(\sin \left(\frac{b^2}{4 a}\right) C\left(\frac{2 a
   x-b}{\sqrt{a} \sqrt{2 \pi }}\right)-\cos \left(\frac{b^2}{4 a}\right)
   S\left(\frac{2 a x-b}{\sqrt{a} \sqrt{2 \pi }}\right)\right)}{\sqrt{a}}$$ I suppose that you have all pieces to finish your problem.
