I am reading the book on Monte Carlo by Sobol (A Primer for the Monte Carlo Method). In the section on Importance Sampling, he writes:
$I = \int_a^b g(x) \: dx$
"to compute this integral, we could use a random variable $\xi$, defined over the interval (a,b) with density $p(x) > 0$. In any case":
$M\begin{equation} \tag{equation 1} \eta = M(g(\xi)/p(\xi)) = I \end{equation}$
where $\eta = g(\xi)/p(\xi)$
and:
$\begin{equation} \tag{equation 2} M\eta = \int_a^b { g(x) \over p(x) } p(x) \: dx = I\end{equation}$
In Sobol's book, M stands for the expected value and D stands for variance. He then writes: "the variance $D\eta$, and hence, the estimate of this error of approximation depend on what variable $\xi$ we use, since:
$\begin{equation} \tag{equation 3} D\eta = M(\eta^2) - I^2 = \int_a^b { g^2(x) \over p(x) }\:dx - I^2 \end{equation}$
Let us prove that $D\eta$ is minised when the density $p(x)$ is proportional to $|g(x)|$. We will use an inequality well known in analysis:
$\begin{equation} \tag{equation 4} \left( \int_a^b |u(x)v(x)|\:dx\right)^2 \leq \int_a^b u^2(x) \: dx \int_a^b v^2 (x) dx \end{equation}$
If we set $u = g(x)/\sqrt{p(x)}$ and $v = \sqrt{p(x)}$, then we obtain:
\begin{equation} \tag{equation 5} \left( \int_a^b |g(x)|\:dx\right)^2 \leq \int_a^b { g^2(x) \over p(x) } \: dx \int_a^b p (x) dx \\ = \int_a^b {g^2(x)\over p(x)\:dx} \end{equation}
It follows from eq. 3 and eq. 5 that:
$D\begin{equation} \tag {equation 5} \eta \geq \left( \int_a^b |g(x)dx\right)^2 - I^2\end{equation}$
My questions:
what would be the proof of equation 1? Why taking the expected value of g(x)/p(x) is equal to I? I don't understand that statement. I understand equation 2 because in the integral we divided g(x) by p(x) but also multiply by p(x). So the two terms cancel out. But I don't understand eq. 1.
I know the formula for computing variance:
$Var(X) = E[X^2] - E[X]^2.$
which is where eq. 3 comes from. However I don't understand the first term on the right inside. Why would $E[X^2] = g^2(x)/p(x)$ and not $E[X^2] = g^2(x) / p^2(x)$?
Where is equation 4 coming from. Sobol says it's a well known inequality but I would like to know where it comes from.
Finally, I really don't understand the last part of the demonstration. What's the meaning of $\left(\int_a^b g(x) \:dx \right)^2 - I^2$ and why is the variance $D\eta$ greater than this equation since I thought we were trying to show that the variance of the general MC estimator was less than the basic MC estimator. Furthermore, as pointed by Sobol, isn't importance sampling only working if p(x) is more or less a function proportional to the integrand g(x). In other words, is it possible to come with a general proof that importance sampling works since it depends highly on the choice of the pdf.
Sorry lots of questions, but this seems critical for the understanding of the method, and hope this post will help other people after me.
Thanks for your help, spending time reading/answering the post and sharing your knowledge.