On rate of decay of integrals (fourier transform) Given, $f(x)$ is a monotonically decreasing function, and $f(x) \geq 0$ for all $x \in [a,b]$. 
Suppose that for $f(x)$ the following holds (Riemann-Lebesgue Lemma):
$$\lim_{n\to\infty} \int_a^b f(x)\sin(nx) = 0.$$
Now lets say that another function $g(x)$ is a periodic function that jumps periodically. For example: $g(x) = \{x\}$ the fractional part of x.  Also it follows the condition that, $0 \leq g(x) \leq 1$ for all $x \in [a,b]$.
If by the R-L lemma
$$\lim_{n\to\infty} \int_a^b g(x) f(x)\sin(nx) = 0.$$
My question is this:

  
*
  
*Can we say that the second expression tends to zero faster than the first?
  
*How do we determine or compare this rate of decay so that we can get an answer to question 1?

 A: Let me offer you three examples. (For simplicity of computation, I suggest using $\sin (n\pi x)$ instead of $\sin (nx)$, but the results are morally the same. 
Case 1
Take $a = -1$, $b = 0$, and $f(x) = -x$, while $g(x) = \{x\}$. You can compute the Fourier transform by explicitly integrating by parts. You will find
$$ \int_{-1}^0 f(x) \sin(n\pi x) \mathrm{d}x = \frac{(-1)^{n+1}}{n\pi} = O(n^{-1}) $$
while
$$ \int_{-1}^0 g(x) f(x) \sin(n\pi x) \mathrm{d}x = O(n^{-2}) $$
so indeed the latter decays faster. 
Case 2
Take $a = -1$, $b = 0$ as before, and $f(x) = -x$. Now set $g(x) = \{x+1/2\}$. The first integral is the same, the second evaluates to
$$ \int_{-1}^0 g(x) f(x) \sin(n\pi x)\mathrm{d}x = - \frac{1}{2n\pi} \left( \cos \frac{n\pi}{2} - (-1)^n\right) + O(n^{-2}) $$
The first term in the right hand side is $-\frac{1}{2n\pi}$ if $n$ is odd; $0$ if $n = 4k$, and $-\frac{1}{n\pi}$ if $n = 4k+2$. So the second integral decreases no faster and no slower than the first. 
Case 3
Take $a = -2$, $b = 0$ still. Now $f(x) = 1$ and $g(x) = \{x\}$. A simple computation shows that
$$ \int_{-2}^0 \sin(n\pi x) \mathrm{d}x = 0 $$
But
$$ \int_{-2}^0 \sin(n\pi x) g(x) \mathrm{d}x = \begin{cases} 
0 & n = 2k+1 \\
- \frac{2}{n\pi} & n = 2k\end{cases} $$
so the latter integral clearly decays slower.  

So what's going on here? Like user15464 mentioned in his comment, the rate of decay for Fourier coefficients $\hat{f}$ is directly related to the smoothness of the original function $f$. In Case 1, we chose $f,g$ such that while each of them has a jump discontinuity, their product extends to a continuous function on $\mathbb{R}$, and hence we improve the decay slightly. In Case 2, the product $fg$ is still discontinuous, and our computation confirms that it decays just as fast (slow) as just $f$. In Case 3, $f$ can be extended to a continuous function on the circle; but $fg$ cannot. And our computation show that the corresponding Fourier coefficients reflect that. (The simple form of the decay ($n$ to some integer power) is due to my trying to give explicit functions whose Fourier transform are easy to compute. One can even get decay rates slower than $n^{-1}$.)
But while smoother functions (those admitting more continuous derivatives) gives more decay, things get more complicated once you allow yourself a function that is only Lebesgue integrable. In fact, it is possible to show that given any sequence of positive real numbers $(\lambda_n)$ such that $\lim \lambda_n = 0$, there exists a continuous function $h$ such that its Fourier coefficients satisfy $\limsup_{n\to\infty} \frac{|\hat{g}(n)|}{\lambda_n} > 1$. (See Exercise 18, Chapter 3, Stein & Shakarchi Fourier Analysis for hints on how to prove this.) (So even the distinction between continuous and discontinuous functions are not as simple as the examples above may suggest.)
Thus starting with $f$, assume now $f > 0$, then choosing $\lambda_n$ such that $\limsup \lambda_n / |\hat{f}(n)| = +\infty$, and defining $g = h / f$ you find a function $g$ such that the Fourier transform of $fg = h$ decays much slower than that of $f$. 
A: You can't say for sure, which of the two integrals tends to zero faster. It depends on the functions you choose. You must find the limes of the quotient of these integrals. For determining it for specific functions you could use L'Hopital's method. It says that $$\lim(f(x)/g(x))=\lim(f'(x)/g'(x)).$$ 
You can use it since you can differentiate the expression inside the integral instead of differentiating the integral. So you can differentiate your function instead of computing the integrals.
