Assume that $X$ is a discrete random variable with support $\left\{x_{1},x_{2},...,x_{n} \right\}$. Consider $\Pr \left[ f \left( X \right) <z \right] $. According to the Law of Total Probability, $\Pr \left[ f \left( X \right) <z \right]=E\left[ \Pr \left[f\left( X \right) <z|X \right] \right]= \sum_{i=1}^{i=n} \Pr \left[ f\left( X \right) <z|X=x_{i} \right] \Pr\left[X=x_{i} \right].$

The question is the following: can we plug $x_i$ directly into $f(X)$, to get $\sum_{i=1}^{i=n} \Pr \left[ f\left( x_i \right) <z|X=x_{i} \right] \Pr\left[X=x_{i} \right]$?

If the function $f(X)=X$ then the equations above work just fine. However, if $f(X)=var(X)$ then we end up with nonsense. Indeed, $\Pr \left[ var \left( X \right) <z \right]=\sum_{i=1}^{i=n} \Pr \left[ var\left( X \right) <z|X=x_{i} \right] \cdot \Pr\left[X=x_{i} \right]=\sum_{i=1}^{i=n} \Pr \left[ var\left( x_{i} \right) <z|X=x_{i} \right] \cdot \Pr\left[X=x_{i} \right] =\sum_{i=1}^{i=n} \Pr \left[ 0 <z|X=x_{i} \right] \cdot \Pr\left[X=x_{i} \right]=\sum_{i=1}^{i=n} 1 \cdot \Pr\left[X=x_{i} \right]=1, $ if $z \geq 0$ or, $=0,$ if $ z <0$

So why the Law of Total Probability ( or its version) does not work in the case of variance?

  • $\begingroup$ Cannot follow all of the last step. Since $Var[X]$ is a constant, $\Pr\{Var[X] < z\}$ is either $1$ or $0$. And when you apply the Law of Total Probability on this, it stills holds, so we will not end up with nonsense. $\endgroup$ – BGM Sep 15 '19 at 3:01
  • $\begingroup$ The $\Pr(Var(X)<z)=1$ if $Var(X) <z$, but by applying the LTP we have this probability equal 1 when $z \geq 0$. Does it make sense? $\endgroup$ – Bob Sep 15 '19 at 5:00
  • $\begingroup$ If $Var[X] < z$, then $\Pr\{Var[X] < z\} = 1$ and $\Pr\{Var[X] < z|X = x_i\} = \Pr\{Var[X] < z\} = 1$ so they are the same. The key is $\Pr\{Var[X] < z|X = x_i\} = \Pr\{Var[X] < z\}$ so it guarantee the result make sense. I am not sure how you obtain your claimed result. $\endgroup$ – BGM Sep 15 '19 at 15:13
  • $\begingroup$ If $Var(X)=2$ and $z=1$ then $\Pr(Var(X)<z)=\Pr(2<1)=0$, but if we apply LTP then we get $\Pr(Var(X)<z)=1$ since $z>0$, to see it just follow the derivation with LPT (as in the question). Does it make sense? $\endgroup$ – Bob Sep 15 '19 at 15:47
  • $\begingroup$ As said, the key is $\Pr\{Var[X]<z|X=x_i\}=\Pr\{Var[X]<z\}$ and $\Pr\{Var[X]<z|X=x_i\} \neq \Pr\{0 < z\}$ in general. The event has a probability of either $1$ or $0$ and thus independent to any other event, including itself. If you plug in the above example, $\Pr\{2 < 1\} = 0$, and $\Pr\{2 < 1| \text{anything}\} = 0$. $\endgroup$ – BGM Sep 15 '19 at 18:21

Your mistake is in saying that on the event $\{X=x_i\}$ we have $\text{Var}(X)=\text{Var}(x_i)$. That is not the case. $\text{Var}(X)$ is simply a constant.

A random variable $X$ can be described as a function. Here, perhaps $X$ is a function from $\{1,2,\dots, n\}$ with $X(i)=x_i$. This is a deterministic function, but we think about the input as being random. So the probability that $X$ takes on the value $x_i$ is the probability that $i$ is the input to the function $X$.

Now when we write $f(X)$ this is really a composition of functions. I.e., on input $i$ we have $f(X)=f\circ X(i)=f(x_i)$. So $f(X)$ is itself a random variable, that is, another function with domain $\{1,2,\dots,n\}$. When we apply $f$ to the random variable $X$ we get out another random variable $f(X)$. Keep in mind that this $f$ must have domain containing the range of the random variable $X$ (in this case, $\{x_1,\dots,x_n\}$).

However, when we write $\text{Var}(X)$, this is not a composition of functions. There is not input to $\text{Var}(X)$. Rather, $\text{Var}$ is a function from the set of random variables (with finite second moment) to $\mathbb{R}$. So $\text{Var}$ takes in a random variable $X$ and spits out a number $\text{Var}(X)$. Notice that $\text{Var}$'s domain does not contain the range of $X$, since $\text{Var}$ takes in random variables, not numbers. So we cannot say $f=\text{Var}$ and apply the formula you derived for $f$.

The reason this may be confusing is because we don't like to write random variables as functions, since it's easier to just think of them as numbers that take on random values. Hence the notation gets overload, so that $f(X)$ denotes a random variable while $\text{Var}(X)$ denotes a number.

  • $\begingroup$ Thanks for the response, but I cannot buy it. A number, which is a constant in this case, is also a random variable, though a specific one. Please, look at the definition of a random variable and try to see that a constant fits it pretty well. $\endgroup$ – Bob Sep 14 '19 at 17:11
  • $\begingroup$ Also, a random variable is not a function (or map) from numbers, like 1,2,3 etc, but a map from elements of a sample space to numbers, so your writing that "Here, perhaps X is a function from $\{1,2,3...,n\}$ " seems to be not accurate. $\endgroup$ – Bob Sep 14 '19 at 17:18
  • $\begingroup$ Actually, you'll see in the very first paragraph on the random variable wikipedia page that formally a random variable is a "measurable function." And you are correct that a constant is also a random variable. In any case, the important distinction is that $f$ is a function that takes in numbers and spits out numbers, while $\text{Var}$ is a function that takes in random variables and spits out numbers. Thus $f(X)$ is a random variable, while $\text{Var}(X)$ is a number. $\endgroup$ – kccu Sep 14 '19 at 17:36
  • $\begingroup$ If we take an expectation instead of variance then the formula seems to work all right, though expectation takes in a random variable and "spits out" a number..... $\endgroup$ – Bob Sep 14 '19 at 18:04
  • $\begingroup$ Also, we can consider $var(X)$ as a function from $R^n$ to R. $\endgroup$ – Bob Sep 14 '19 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.