3
$\begingroup$

I am studying Probability and Monte Carlo methods, and it feels that the more I study the less I truly understand the theory. I guess I just confuse myself now. So the expected value of a random variable X is:

$E[X] = \int x_i p(x_i)\:dx$

where X is is random variable and p(x) is the pdf of X. This sort of makes sense to me especially of you think in terms of discrete random variable. If an outcome is more likely to be drawn than others, then the pdf for this outcome $x_i$ accounts for this.I have a problem when this apply to a function. So we have:

$E[f(x)] = \int f(x_i) p(x_i) \: dx$

My understanding here is that f can be any sort of function we like. $x_i$ is a possible value from the random variable X and $p(x_i)$ is the probability distribution. Because $x_i$ is random $f(x_i)$ is random. So f(x) is a function of the random variable X.

Now what I don't understand with this: doesn't the result of the expected value E[f(x)] depends on the choice of the pdf? For example of you have a simple function such as f(x) = x (imagine the integration domain is [0,1]), if you choose a uniform pdf or a gaussian distribution, wouldn't the result of E[f(x)] be different?

I am sure I am just mixing in my mind "simple" concepts. I am probably not thinking this problem the right way. If anybody could straighten that up for me, it would be great.

EDIT

Thank you for your answers. So it seems my question wasn't clear enough and complete. From the answer you gave me, I understand that:

  • X is distributed according to a give distribution (in other words, X and its PDF are interdependent).
  • therefore indeed changing X, implies that the distribution has changed as well
  • and yes, E[f(X)] is likely to be different for different Xs.

I think my confusion comes partly from the fact this PDF plays a role in the computation of the Monte Carlo integral and particular the general form in which the integrand is divided by the PDF:

$E[f(X)] \approx { 1 \over N } \sum_{i=0}^{N-1} { f(x_i) \over p(x_i) }.$

where $x_i$ is a sequence of random numbers drawn from a random variable X with distribution p(X). We are trying to approximate:

$F = \int f(x)\:dx.$

We know the result of this MC integral converges in probability to the expected value E[f(X)]. So according to the three points I listed above, wouldn't changing X and its PDF give a different result for E[f(x)] when MC integration is used?

When you measure the area "under" (a way of interpreting what an integral is) the curve, that area is constant. So if we get a different E[f(x)] for something that should always be the same, what I am missing?

In other words, in a lot of books $E[f(X)] = \int f(X) p(X) \: dx$ is presented as "the formula" to calculate the expected value of f. This seems like misleading to me. Should it be more accurate to say "the expected value of f given the particular random variable X with PDF p(X)"?, knowing that if we change X we will get a different E[f(X)].

$\endgroup$
  • $\begingroup$ Usually the random variable X has an underlying distribution, so if you change the distribution, it's not X as originally defined. $\endgroup$ – Silynn Jun 20 '14 at 23:16
  • $\begingroup$ @Silynn. So you are saying that the x's are distributed according to the distribution, if I understand what you are saying. My question was then that if you change X (and therefore it's associated distribution), then E[F(x)] changes. That's what I would like to know. $\endgroup$ – Marc Ourens Jun 21 '14 at 7:22
  • $\begingroup$ Yes, E[f(X)] changes when you change the distribution. $\endgroup$ – Silynn Jun 21 '14 at 7:25
  • $\begingroup$ Thank you Silynn, I made my question more precise then. See the EDIT section. $\endgroup$ – Marc Ourens Jun 21 '14 at 7:44
1
$\begingroup$

Let me add a little bit to what's already been said.

In your first example, $E[X] = \int x_i p(x_i)\:dx$, you are looking for the expected value of $X$. Where did the $p$ come from? The original problem would have been to find $E[X]$, given that $X$ has the distribution $p$. Often, $p$ is not explicitly stated when asking the question about the expectation. However it probably was stated as the pdf when $X$ was first brought into the problem.

The same thing is true of your second example. In that case, in order to even ask the question about the expectation of $f(x)$, you would have to have been given the distribution of $x$.

By the way, in this case it would generally be written $f(X)$, with the $X$ in upper case to indicate it's a Random Variable. Generally, the lower case letter is then used, as you did use it, in your integral.

Also, with a continuous Random Variable you wouldn't use the subscript $i$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for correcting the formula. But my question which I believe Dommrobo has answered is the same. Then if you change X and therefore change its PDF, then the result of E[f(X)] changes(?) $\endgroup$ – Marc Ourens Jun 21 '14 at 7:23
1
$\begingroup$

It seems like what I am looking for and that I am missing is called the Law of the Unconscious Statistician.

http://en.wikipedia.org/wiki/Law_of_the_unconscious_statistician

So it seems to prove that if you know the distribution of X but don't know the distribution of the function f(X), then we can still calculate the expected value of f(x), using:

$E[f(x)] = \int f(x) p(x) \: dx.$

I still need to understand the proof and find some simple example that would help me understanding the process better. Could someone please confirm this is the right answer?

It's interesting because that still seems to invalidate the fact that changing the random variable changes E[f(x)]. This law seems to suggest the opposite. E[f(x)] will always be the same, which makes more sense to me, because that's what we are trying to calculate thus it has no reason to change. It is just the way we compute it that differs and we can use this law, if we don't know the distribution of f(X) in the first place.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

The choice of pdf does affect the value of $E[f(x)]$. Take a trivial example: suppose you have $p(.5)=1$ and $p(x)=0$ everywhere else. Then your expected value ought to be $f(.5)$. You can do this for any single point. Knowing the pdf of the space is necessary to calculate the expected value.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you. I think my confusion comes partly that this PDF plays a role in the computation of the Monte Carlo integral and particular the general form in which the integrand is divided by the PDF. Since the result of a MC integral converges in probability to the expected value E[f(X)], wouldn't changing X and its PDF give a different E[f(x)]. When you measure the area "under" (the integral) the curve wouldn't that be constant though? So we do get a different E[f(x)] for something that should always be the same? That's what I don't understand I think. $\endgroup$ – Marc Ourens Jun 21 '14 at 7:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.