I am still progressing in my learning of probability and monte carlo method. I understand a basic MC estimator can be written as:

$$\bar x = { 1 \over N } \sum_{i=1}^N f(x_i) \approx E[f(x)]$$

I understand this can be seen as an approximation of $E[f(x)]$. Where it assumes that $x$ and $x_i$ are distributed according to the same probability distribution function $p$. I understand these things so far. Now what I don't understand in the book I am reading is the following thing: it computes the expected value of $\bar x$ and I am not sure how to interpret that? This is confusing because $\bar x$ is already an approximation of the mean $E[f(x)]$. So I am not too sure what the "mean of the approximation of the mean $E[f(x)]$" means. It's written in the book that:

$$E[\bar x] = E[f(x)]$$

The way I interpret this is the following and I'd like to know if this is accurate. The sample mean $\bar x$ is just an approximation. Thus if we compute the mean of this approximation (which comes back to say that you compute an infinity of these sample mean and take the average value), then the result is actually identical to the mean $E[f(x)]$. Wouldn't that be the same though than computing the sample mean $\bar x$ where the number of samples N goes to infinity?

Where I am also confused is that in the book it says that:

$$E[\bar x] = {1 \over N} \sum_{j=1}^N E[f(x_j)]$$

I just don't get where the $E[f(x_j)]$ on the right inside comes from. Why would computing the mean of $\bar x$ implies a sum of mean $E[f(x_j)]$? It's probably simple but I can't make sense of what's going on really.

Thank you.

  • $\begingroup$ Which book is that? $\endgroup$ – leonbloy Sep 3 '13 at 23:26
  • $\begingroup$ It's a book about CGI "Advanced Global Illumination" by P. Dutre. $\endgroup$ – Marc Ourens Sep 4 '13 at 11:25

It is true that if $Y_i$ have finite means, then $E[\sum a_iY_i] = \sum a_i E[Y_i]$. This is known as linearity of expectation. In your case, the author took the definition of $\overline{x}$ $$ \overline{x} = \frac{1}{N}\sum_{i=1}^N f(x_i) $$ and took the expectation of both sides, so $$ E[\overline{x}] = E[\frac{1}{N}\sum_{i=1}^N f(x_i)] = \frac{1}{N}\sum_{i=1}^N E[f(x_i)]. $$

The reason the author does this is because, although $\overline{x} \approx E[f(x)]$ is intuitively clear, this statement has no mathematical meaning. However, the formula for $E[\overline{x}]$ above provides useful information on how good the approximation is.

  • $\begingroup$ Thank you. I made some progress but I am still not sure how I should interpret the expected value of the sample mean. If I flip a coin 5 times and get 0,0,1,0,1, the sample mean is 2/5. Should I see the expected value of the sample mean as if I was repeating the experiment where I flip the coins 5 times a very large number of times to find that the value is going to converge to 0.5? (assuming the coin is not loaded, thus probability to get either 0 or 1 is 1/2)? Thank you. $\endgroup$ – Marc Ourens Sep 3 '13 at 23:50
  • $\begingroup$ @MarcOurens You are thinking along the right lines. What you are describing is the result of the law of large numbers which tells you that if $Y_i$ represents the result of an independent copy of some experiment, and $E[Y_i]=\mu$ finite for all $i$, then $\frac1N \sum Y_i$ converges almost surely (as well as some other kinds of convergence!) to the constant function $\mu$. en.wikipedia.org/wiki/Law_of_large_numbers $\endgroup$ – nullUser Sep 4 '13 at 0:11
  • $\begingroup$ Yes I read about this and it's good you point it out again. But when you say I think along the right line, is that accurate (enough) then to interpret the "expected value of the sample mean" the way I described it in the comment or would there be a better interpretation/definition of it? How would you explain it to someone. Thank you. $\endgroup$ – Marc Ourens Sep 4 '13 at 0:18
  • $\begingroup$ I think I got it but it would be great if you could confirm. In fact I was reading about the mean on Wiki (en.wikipedia.org/wiki/Mean) and saw the difference between the sample mean and the population mean. In another document, I found that $E[\bar x] = \mu$ where $\mu$ is the population mean. So the expected value of the sample mean is actually equal to the population mean where the population mean "is equal to the arithmetic mean of the given property while considering every member of the population". The former ($\bar x$) is an approximation, the latter ($\mu$) is exact. Yes? $\endgroup$ – Marc Ourens Sep 4 '13 at 0:37
  • $\begingroup$ @MarcOurens I'm a little fleeting in terms of confirming your definition. This is because in probability one very quickly runs into terms which are either extremely technical or based on philosophical grounds. The least convoluted and technical way I know how to say it is that expectation is an averaging operation. In fact, it is an integral. Think of $\frac{1}{b-a}\int_a^b f(x)dx$. You recognize this as the average of $f$ over $[a,b]$. The definition of= $E[X]$ is $\int_{\Omega} X(\omega)dP(\omega)$ where here we are using the Lebesgue integral. $\endgroup$ – nullUser Sep 4 '13 at 18:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.