7
$\begingroup$

I've previously asked the question on Stats SE, but I guess it fits the Math SE better. Is it possible to rigorously derive the formula for expected value of continuous random variable starting with expected variable in discrete case, i.e.

$$E[X] = \sum_{i=1}^{n}p_i x_i$$

to obtain

$$E[X]=\int_{-\infty}^{\infty}xf(x)dx$$

When formulating the definition for continuous case, the intention was, I believe, to make it 'equivalent' to the discrete case. So for example, I'd like $E[X]$ of a continuous random variable $X$ to be equal to the sum of every possible value of $X$ times probability of that particular value.

The problem is that the probability of any particular value of $X$ is $0$, and the expected value calculated that way would always be $0$. But some people tried to convince me that it's possible overcome these issues with help of Lebesgue integral. Could anyone explain intuitively how is that possible? I'm convinced that no matter what integration we use, we cannot somehow magically assign non-zero probabilities to single values the random variable $X$ might take. They will always be $0$!

Or maybe there's no magic involved and best we can do is work with infinitely thin intervals of $X$? From what I've managed to find out (I don't have time to study all measure theory and Lebesgue integration at the moment to figure it out on my own) it's about approximating the original continuous random variable $X$ with step function, and the more steps there are, the better is the approximation. But still, all we have is calculating probability of intervals (they are infinitely thin though, but they are intervals anyway).

The fact that something gets closer and closer to original function in the limit doesn't mean it behaves the same as the original function (check the very popular example here).

In the 'very popular example' above, even though the curve approaches the circle, its length will never be the same as the perimeter of that circle. Similarly, here, in 'The Riemann-Stieltjes integral: intuition' part. they say the discrete r.v. $X_n$ converges to continuous r.v. $X$, as $n \to \infty$, in the limit it becomes the same variable. So it's important to ask whether it's reasonable to expect the approximation of the cont. r.v. $X$ to behave the same as the original random variable $X$, even if in the limit the are 'indistinguishable' in the limit, whatever that word means in mathematics. The curve and circle are indistinguishable too, but still have different properties.

So apparently it's that the continuous case is not derived from discrete case, but is a generalization of the discrete case. I guess in every mathematical theory, there is no such thing as 'the only correct' generalization, so the contunous formula could look different. If you claim this is the only 'valid one', then shouldn't we call it a derivation of the formula?

$\endgroup$
  • $\begingroup$ If your probability space is an interval, then you can definitely carry out this approximation process with the Riemann integral. (If your random variable is $X=X(\omega)$, then the "probability space" is the space in which the $\omega$'s live). $\endgroup$ – Giuseppe Negro Sep 16 '15 at 21:05
  • $\begingroup$ I have added in the comment of my answer the computation for a discrete random variable. Hope its useful :) $\endgroup$ – Calvin Khor Sep 17 '15 at 20:21
  • $\begingroup$ " I'd like $E[X]$ of a continuous random variable $X$ to be equal to the sum of every possible value of $X$ times probability of that particular value" A more basic-fundamental statement would be "I'd like the probability that the continuous random variable $X$ takes any value (which is $1$) to be equal to the sum of the probabilities of each particular value." ("The problem is that the probability of any particular value of $X$ is $0$")... $\endgroup$ – leonbloy Sep 24 '15 at 14:45
14
+100
$\begingroup$

You are slightly missing the point. The 'right way' is not to define continuous r.v. expectation to match the discrete case, but to find a structure that they both share. Lebesgue integration defines both continuous and discrete expectations at the same time, basing everything on the probability set-function $P$, which is normally called a probability measure, or just a probability. $P$ measures the size of sets in the event space, e.g. $P(\text{$X$ is a heads}) = 1/2$. We can also talk about the induced probability measure(or law, or distribution) of $X$, $\mu_X = P\circ X^{-1}$. The same example is $P(X^{-1}(\text{heads})) = 1/2$.

Suppose a random variable $X$ takes only a finite number of values, $X = \sum_{i=1}^N x_i\Bbb 1_{A_i}$. This is termed a 'simple function'. Here $\Bbb 1_A$ is the function $$ \Bbb 1_A(ω) := \begin{cases} 1 & ω ∈ A \\ 0 & ω \not∈ A \end{cases}$$ and it follows that the sets $A_i = [X = x_i]$ as long as the $A_i$ are disjoint sets.

We then define expectation, the integral of $X$ with respect to the probability $P$ to be

$$\Bbb E X:= ∫_Ω X(ω) dP(ω) := \sum_{i=1}^N x_i P(A_i) $$

($∫_Ω X(ω) dP(ω)$ is also written $∫_Ω X dP, ∫_Ω X(ω) P(dω),\dots$)

Note that if the probability of a singleton e.g. $A_i = \{\text{heads}\}$ is positive, $P(\{\text{heads}\}) > 0$, this means we will assign a single point some positive probability with this integration method, which I believe solves one of your problems.

For a random variable that can take more than finitely many values, we approximate by simple functions, and take limits, $$\Bbb E X:=∫_Ω X(ω) dP(ω) := \lim_{n→∞}∫_Ω X_n(ω) dP(ω)$$

Once this is out of the way, we can then distinguish discrete and continuous (or combinations of) random variables by looking at their distribution (induced measures). For a discrete $X$, we would have the distribution $\mu_X = \sum_{i=1}^∞ P(X=x_i)δ_{x_i}$, where $δ_x$ are the Dirac point masses,

$$ \Bbb δ_x(A) := \begin{cases} 1 & x ∈ A \\ 0 & x \not∈ A \end{cases}$$

It is not hard to check that

$$\Bbb E X = ∫_{-∞}^∞ x d\mu_X(x)$$

In the case of a continuous random variable, this means that we have a density function $f$ such that $f(x) dx = d\mu_X$. Again, it is just a little harder to check that

$$\Bbb E X = ∫_{-∞}^∞ x d\mu_X(x)$$


From comments-

is that structure unique, or maybe it is possible to find another structure they would both share, with different formula for continuous case?

That question is very broad, and I don't know if one day we will discover a better foundation for Probability than measure theory. However, another good thing about Lebesgue integration is that we have some very powerful convergence theorems: e.g.

Monotone Convergence Theorem. If $f_n,f$ are $\mu$-measurable(i.e. 'nice') functions such that $f_n$ increases to $f$ i.e. $f_n(x)\uparrow f(x)$ for every $x$, then $$∫_Ω f_n \ d\mu \uparrow ∫_Ω f \ d\mu$$

and

Dominated Convergence Theorem. If the measurable functions $f_n,f$ are such that $f_n$ converges pointwise to $f$ and there is a measurable $g>0$ such that $∫_Ω g \ d\mu <∞ $ and $|f_n|<g$, then $$∫_Ω f_n\ d\mu → ∫_Ω f\ d\mu $$

IMO, these results are very intuitive, and work for any measure (discrete and continuous!) so if we somehow had a different way to integrate that violates one of these, I think would view those as the 'wrong' way to integrate. But if the new integration method doesn't violate these, is it really a different formula?

$\endgroup$
  • 2
    $\begingroup$ @user4205580, for a discrete random variable $X$, we have $$ ∫_{\Bbb R} x d\mu_X(x) = ∑_1^∞ P(X=x_i) ∫_{\Bbb R} x \ dδ_{x_i} $$, and $ ∫_{\Bbb R} x \ dδ_{x_i} = x_i$. Here we have used that the integral against a sum of measures is defined as the sum of integrals against each measure, and appropriate limit theorems. $\endgroup$ – Calvin Khor Sep 17 '15 at 20:18
  • $\begingroup$ I realised there's one issue here - 'The 'right way' is not to define continuous r.v. expectation to match the discrete case, but to find a structure that they both share.' - is that structure unique, or maybe it is possible to find another structure they would both share, with different formula for continuous case? $\endgroup$ – user4205580 Sep 20 '15 at 15:20
  • 2
    $\begingroup$ What I meant was if the formula for continuous case is derived from discrete case, or maybe it's a generalization? In the beginning of your original answer you seem to say it's the latter. It's a generalization, and we use Lebesgue integral to capture both cases in one formula. But mathematicians could have generalized in in a different way, right? If not, then we should call it really a 'derivation' of the formula for continuous case. $\endgroup$ – user4205580 Sep 20 '15 at 16:02
  • 2
    $\begingroup$ This is the source of my confusion. A few people I know tried to convince me the E[X] for contunuous $X$ can be derived from discrete case and I didn't feel comfortable with it. Apparently they were wrong. $\endgroup$ – user4205580 Sep 20 '15 at 16:18
  • 2
    $\begingroup$ I wouldn't agree with that. I guess there are many counter-intuitive theorems. Does it necessarily mean the definitions used to state them are wrong? I really doubt it. The problem with $\pi = 4$ is that it's based on our intuition which apparently says 'if two function converge, they necessarily behave the same and all their properties are the same. Say we have $f(x)$ and $g(x)$ and they converge. It doesn't mean $h(f(x))=h(g(x))$ is true in the limit, for every $h$ function you can come up with. We start with definitions and axioms and can't really judge them by theorems we get. $\endgroup$ – user4205580 Sep 20 '15 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.