For (a):
For each $\newcommand\abs[1]{\left\lvert#1\right\rvert}a>0$, we'd prove that
$$\lim_{x\to+\infty}\int_a^\infty xF(xt)\cos tdt=0$$
(as Macavity has suggested, the improper integral converges out of Dirichlet's test).
Fix $x>0$, take arbitrary $A>a$, we have (by mean value theorem)
$$\int_a^AF(xt)\cos tdt=F(ax)\int_a^c\cos tdt+F(Ax)\int_c^A\cos tdt$$
where $c\in[a,A]$.
Therefore
$$\abs{\int_a^AF(xt)\cos tdt}\le 2F(ax)\int_{-\pi/2}^{\pi/2}\cos tdt$$
for all $A>a$, thus
$$\int_a^\infty F(xt)\cos tdt=o\left(\frac1x\right)\qquad x\to+\infty$$
For (b):
We need to show that
$$\lim_{x\to+\infty}x\int_0^{\pi/2}(F(xu)-G(xu))\cos udu=0\tag{*}$$
given that
$$\lim_{n\to\infty}n\int_0^{\pi/2}(F(nu)-G(nu))\cos udu=0\tag1$$
(for the virtue of (a), we could replace $\int_0^\infty$ by $\int_0^{\pi/2}$)
Suppose $n\le x<n+1$, we write
$$x(F(xu)-G(xu))=\frac xnn(F(nu)-G(nu))-x(F(nu)-F(xu))+x(G(nu)-G(xu))\tag2$$
By (*), we have
$$\lim_{x\to+\infty}\frac xn\cdot n\int_0^{\pi/2}(F(nu)-G(nu))\cos udu=0\tag3$$
The second term could be split into two parts
$$0\le x(F(nu)-F(xu))=(x-n)F(nu)+(nF(nu)-xF(xu))\le F(nu)+(nF(nu)-xF(xu))$$
Since $zF(z)\to0$ as $z\to+\infty$, we choose $\Delta>0$ such that $F(z)\le1/z$ whenever $z\ge\Delta$, therefore
$$\int_0^{\pi/2}F(nu)\cos udu=\int_0^{\Delta/n}+\int_{\Delta/n}^{\pi/2}$$
where
$$\int_0^{\Delta/n}\le F(0)\int_0^{\Delta/n}\cos udu\to0$$
as $x\to+\infty$ and
$$\int_{\Delta/n}^{\pi/2}\le\frac1n\int_{\Delta/n}^{\pi/2}\frac{\cos u}udu\le\frac1n\int_{\Delta/n}^{\pi/2}\frac{du}u=O\left(\frac{\ln n}n\right)\to0$$
Rewrite
$$\int_0^{\pi/2}(nF(nu)-xF(xu))\cos udu=\int_0^{n\pi/2}F(v)\left(\cos\frac vn-\cos\frac vx\right)dv-\int_{n\pi/2}^{x\pi/2}F(v)\cos\frac vxdv$$
Note that
$$0\le-\int_0^{n\pi/2}\le 2F(0)\int_0^{n\pi/2}\sin\frac12\left(\frac vn-\frac vx\right)=O\left(\frac1x\right)$$
and
$$\int_{n\pi/2}^{x\pi/2}\le\frac{(x-n)\pi}2F(n\pi/2)\to0$$
Hence
$$\int_0^{\pi/2}x(F(nu)-F(xu))\cos udu\to0\tag4$$
Similarly,
$$\int_0^{\pi/2}x(F(nu)-F(xu))\cos udu\to0\tag5$$
(*) follows from (2),(3),(4),(5).