By way of disclosure, let me start by saying that I am not a mathematician. However, I came to peruse Kyle Siegrist's web-based text in probability theory (http://www.randomservices.org); and I a quite intrigued.

Unfortunately, I am currently struggling to reconcile a few of the statements that I came across. Since my struggles might be due to a somewhat more fundamental lack of understanding, I trust you can forgive, if I start my question by summarizing what I believe to have understood so far. Pleasure feel free to skip to the demarcated passage and refer to the introduction only if you feel that I misunderstood some of the more fundamental aspects of probability theory.

I start with a probability space $( \Omega,\mathcal{F},\mathbb{P})$ where $\Omega$ is a finite, countably infinite , or uncountably infinite sample space and $\mathcal{F}$ is a $\sigma$-algebra furnishing a domain for probability measure $$\mathbb{P}\,:\,\mathcal{F}\,\rightarrow\,[0,1].$$ Now, I consider a measurable space $(\mathbb{R},\mathcal{B})$ where $\mathbb{R}$ is the real line and $\mathcal{B}$ is the $\sigma$-algebra generated by the standard Euclidean topology, i.e. the extension of the collection of all open sets in $\mathbb{R}$ to a $\sigma$-algebra. This measurable space allows for the definition of a measurable function $$X\,:\,\Omega\,\rightarrow\,\mathbb{R}$$ such that the preimage $X^{-1}(A)\in\mathcal{F}$ for all $A\in\mathcal{B}$. I suppose that could also be written as $\sigma(X)\subseteq\mathcal{F}$. $X$ is then called a random variable. Utilizing a change of variables theorem, I proceed to define a measure $\mu$ on $(\mathbb{R},\mathcal{B})$ that corresponds to $\mathbb{P}$ such that $$\mu(A)=\mathbb{P}[X^{-1}(A)]\quad\forall\quad A\in\mathcal{B}.$$ Since $X^{-1}(A)\in\mathcal{F}$, $\mathbb{P}(X^{-1}(A))$ exists in $[0,1]$. Hence, $$\mu\,:\,\mathcal{B}\,\rightarrow\,[0,1]$$ and is called the distribution of $X$. From there it is easy enough to show that $\mu$ is, in fact, a probability measure making $(\mathbb{R},\mathcal{B},\mu)$ a probability space. Now, I can make a statement about a random variable that corresponds to an event: $$X(\omega)\in A\quad(A\in\mathcal{B})\quad\text{corresponds to}\quad X^{-1}(A)\in\mathcal{F}.$$Moreover, I can assign corresponding probabilities since $\mu(A)=\mathbb{P}[X^{-1}(A)].$ Finally, I recognize that intervals (on the real line) of the type $$(-\infty,a]$$ are Borel sets such that $(-\infty,a]\in\mathcal{B}$ for all $a\in\mathbb{R}$. Therefore, $\mu(-\infty,a]$ is defined. Equipped with $\mu$, I can then go on to define a distribution function $$F\,:\,\mathbb{R}\,\rightarrow\,[0,1]$$ such that $$F(x)=\mu(-\infty,x].$$ It is reasonably straightforward to show that $F$ is monotonically increasing, right-continuous, and normalized to take values between $0$ and $1$. Moreover, $$\mu(a,b]=F(b)-F(a).$$ and $$\mu\{a\}=F(a)-F(a^{-})\quad\text{where $F(a^{-})$ is the limit from the left}.$$ Hence $F$ is continuous (from the left and from the right) iff $\mu\{a\}=0$ or all $a\in\mathbb{R}$. On the other hand, if $\mu$ has an atom at point $x\in\mathbb{R}$ such that $\mu\{x\}>0$, then $F$ is a step function.

This insight naturally leads to my question.
All of the above did not deal with the question of whether $X$ is a continuous or a discrete random variable. So I like to believe that everything I wrote is valid in both cases.

But: I am not sure how to reconcile that fact that $\mu$ has a domain comprised of uncountable Borel sets with the derivation of the probability mass function (as the Radon-Nikodym derivative of $\mu$ w.r.t the counting measure) for a discrete random variable.

In the continuous case I understood that I can argue as follows: $\Omega$ is necessarily uncountable and moreover the image of $\Omega$ under $X$ is also uncountable. However, it might be an uncountable subset of the reals such as $\mathbb{R}_+$. When I defined the distribution $\mu$ of $X$, though, I did so using as domain all Borel sets in $\mathbb{R}$ - even though X couldn't possibly take values in the negative reals. So what I did above can only be valid, if the Borel sets that are disjoint from the support of $X$ are simply assigned a measure of $0$. (Correct?)

I can also characterize the feature of being a continuous random variable in terms of absolute continuity w.r.t to Lebesgue measure $\lambda$. That is: $$\lambda(\emptyset)=0\qquad\text{and}\qquad \mu(\emptyset)=0$$ $$\lambda\{x\}=0\qquad\text{and}\qquad \mu\{x\}=0$$ where $\mu\{x\}=0$ is only true for continuous random variables.

Question 1: I see how that relates to the distribution function being continuous from the right as well as from the left, but is there a straightforward way to link that to the image of $\Omega$ und $X$ being uncountable?

So, if $\mu\{x\}=0$, the distribution $\mu$ is absolutely continuous w.r.t $\lambda$.
By the Radon-Nikodym Theorem there exists a density (RN-Derivative) such that $$\mu(A)=\int_A\,f\,\text{d}\lambda\quad\forall\quad A\in\mathcal{B}$$

Since, in my example above, every non-singleton subset of the negative reals has zero measure w.r.t. $\mu$ but positive Lebesgue measure, it does follow that the RN-derivative $f(x)$ is simply set to zero, whenever $x\not\in X(\Omega).$ (Correct?)

My struggle is mostly with the discrete case. I would argue that $\Omega$ can be countable or uncountable, but for a random variable $X$ to be discrete $X(\Omega)$ must be countable - so the random variable has countable support.

Moreover, I understood that $X$ is discrete if the distribution $\mu$ is a discrete measure such that $\mu(\mathbb{R}\setminus X(\Omega))=0$ where $X(\Omega)$ is clearly countable. (Correct?)

It is also true that $\#(\emptyset)=0\qquad\text{and}\quad\mu(\emptyset)=0.$ Therefore, the distribution ($\mu$) of $X$ is absolutely continuous w.r.t counting measure ($\#$). And, thus, by the Radon-Nikodym theorem, there exists a density such that. $$\mu(A)=\int_A\,f\,\text{d}\#\quad\forall\quad A\in\mathcal{B}.$$

If $A$ was a subset of the support of $X$ ($A\subseteq X(\Omega)$), i.e. $A$ would only consist of values that $X$ actually might take, then $A$ would be countable, $f$ would be a simple function and $$\mu(A)=\int_A\,f\,\text{d}\#$$ $$\mu(A)=\sum_{x\in A}f(x)\#(\{x\}).$$ Since the count of a singleton is $\#(\{x\})=1$, it would follow that $$\mu(A)=\sum_{x\in A}f(x)$$ and as $\mu(A)=\mathbb{P}(X\in A)$ $$\mathbb{P}(X\in A)=\sum_{x\in A}f(x)$$ such that the RN-derivative of $\mu$ w.r.t $\#$ has the interpretation of the usual probability mass function for discrete random variables.

Question 2: But $\mu$ is defined over the Borel sets. So $\mathbb{P}(X\in A)\,[\,=\mu(A)\,]$ might take the form $\mathbb{P}(a< X\leq b)\,[\,=\mu(a,b]\,]$, which is a valid statement even for a discrete random variable. So, even if I presume that $f(x)=0$ for all $x\not\in X(\Omega)$, how do I come from $$\mu(A)=\int_A\,f\,\text{d}\#\quad\forall\quad A\in\mathcal{B}.$$ to $$\mu(A)=\sum_{x\in A}f(x)$$ since $A$ is not countable and I cannot use countable additivity?

Thank you so very much.

Best wishes,

  • $\begingroup$ I think your question is too long. You don't need to write all the basic definitions; normally people reading this already know them (otherwise we would not be reading this). You should go straight to your point. $\endgroup$ – jpvigneaux Jul 11 '18 at 9:11
  • $\begingroup$ You are absolutely right. I just wanted to make sure that my struggle was not due to any fundamental misunderstanding of the basic definitions on my part. However, I do see that this makes it quite cumbersome to read my question. $\endgroup$ – J.Beck Jul 11 '18 at 20:36

So, if $\mu\{x\}=0$, the distribution $\mu$ is absolutely continuous w.r.t $\lambda$.

That is not true. The condition is necessary and sufficient of $X$ to be continuous in the sense that CDF $F$ is a continuous function, but for the existence of a PDF more is needed: $\mu<<\lambda$ which means that $\mu(B)=0$ must be true for every $B\in\mathcal B$ that satisfies $\lambda(B)=0$. This condition is necessary and sufficient for the existence of a Radon-Nikodym derivative. If $f$ is such a derivative (there is more than one) then in your example indeed also the function prescribed by $x\mapsto f(x)$ for $x\in\mathbb R_+$ and $x\mapsto 0$ otherwise is such a derivative, and in that situation functions with that property are commonly chosen as PDF. Next to that we strive to use a PDF having other nice properties like continuity.

I would argue that $\Omega$ can be countable or uncountable, but for a random variable $X$ to be discrete $X(\Omega)$ must be countable - so the random variable has countable support.

For a discrete random variable it is not necessary that $X(\Omega)$ is countable. For example on $(\Omega=\mathbb R,\mathcal B)$ we can define $\mathbb P$ by setting that $\mathbb P(B)=1\iff0\in B$ and then we can define random variable $X$ by $\omega\mapsto\omega$. That leads to discrete random variable with $\mu(\{0\})=1$, but $X(\Omega)=\mathbb R$ is uncountable. In fact if we are looking for a support then $X(\Omega)$ can be a candidate, but is often not okay. It might also be that $X(\Omega)\notin\mathcal B$ and a support (what exactly is it?...) must be something as a "smallest" set $S\in\mathcal B$ with $\mu(S)=1$. By discrete random variables such a set exists and above it is the set $\{0\}$. Actually $X$ is by definition discrete if a countable set $S\subseteq\mathbb R$ exists with $\mathbb P(X\in S)=1$ or equivalently $\mu(S)=1$ and as shown we do not need $X(\Omega)\subseteq S$ for that. If there are elements $s\in S$ with $\mathbb P(X=s)=0$ then we can just "throw them out" and what is left is again a countable subset with $\mu(S)=1$ and now also $\mu(\{s\})>0$ for every $s\in S$, so that a smaller set with $\mu(S)=1$ cannot exist. Since $S$ is countable we have $S\in\mathcal B$ so that $\mu(S)$ is automatically defined.

In the continuous case we must not search for a smallest set $S$ with $\mu(S)$ simply because it does not exist: if $\mu(S)=1$ then also $\mu(S-T)=1$ whenever $T$ is a countable set. We can look for a nice $S\subseteq\mathbb R$ with $S\in\mathcal B$ and $\mu(S)=1$. In the example you mentioned we can take $\mathbb R_+$, but again it is not necessary that $X(\Omega)\subseteq\mathbb R_+$.

Concerning question 2

Let it be that $S\subseteq\mathbb R$ is countable and serves as support of discrete $X$. Then define $\#A$ to be the cardinality of $A\cap S$. Then $\#$ is a measure on $\mathcal B$ (or even on $\wp(\mathbb R)$). By definition:$$\int_A fd\#=\sum_{a\in S\cap A}f(a)$$

  • $\begingroup$ Thank you very much for your answer. It would seem that I fell victim to an erroneous understanding of the support of a random variable. I understood that the 1-dim. Lebesgue measure is consistent with the notion of the length of an interval in $\mathbb{R}$. All countable subsets of $\mathbb{R}$ therefore have Lebesgue measure 0. As you said, $\mu\ll\lambda$ iff any null set for $\lambda$ is also a null set for $\mu$. So would I be correct, if I were to submit that if $\mu(\emptyset)=0$ and $\mu(S)=0$ for all countable subsets $S\in\mathcal{B}$, then $\mu\ll\lambda$? $\endgroup$ – J.Beck Jul 11 '18 at 20:45
  • $\begingroup$ You are welcome. No, you cannot conclude $\mu<<\lambda$ from that. If $\mu(\{x\})=0$ for every singleton then it can be deduced that $\mu(S)=0$ is true for every countable $S$ on base of $\mu(S)=\sum_{x\in S}\mu(\{x\})$. So actually the condition that $\mu$ takes value $0$ on countable sets is not stronger than the condition that $\mu$ takes value $0$ on singletons, and (again) this is not enough for $\mu<<\lambda$. Have a look here for more info. $\endgroup$ – drhab Jul 12 '18 at 7:41
  • $\begingroup$ You are very welcome. $\endgroup$ – drhab Jul 13 '18 at 8:41

Here a first attempt of an answer: a variable $X:(\Omega, \mathcal F, \mathbb P) \to (\mathbb R, \mathcal B)$ can mix a continuous part and a discrete part. As you said, it is enough to consider the function $F(x):= \mathbb P(X^{-1}((-\infty,x]))$. The variable $X$ is called continuous when $F$ is a continuous function (the law of $X$, say $\mu$, is then a ''continous distribution'', for example: gaussian, exponential, etc.). In general, however, the function $F$ has discontinuities. If $a$ is a discontinuity point of $F$, then $\mu(a) = F(a)-F(a-)>0$ and therefore $\mu$ is not absolutely continuous with with respect to (w.r.t) the Lebesgue measure $\lambda$ (because $\lambda(a)=0$). Therefore, the Radon-Nikodym derivative $d\mu/d\lambda$ does not exists, i.e. the law $\mu$ has no density w.r.t $\lambda$. A ''real-valued discrete random variable'' normally refers to the case when $\mu$ is supported on a countable subset $E\subset \mathbb R$ such that $E$ has no accumulation points. If you consider $E$ as a measured space (with the induced sigma algebra and the counting measure, for example), then $\mu$ has a density w.r.t. the counting measure (any measure on $E$ has a such a density).

  • $\begingroup$ Thank you very much for your answer. Now I see how $\mu\ll\lambda$ is stronger than but implies continuity of $F$ and why one is required to resort to the counting measure to recover a density for a random variable such that $\mu$ is supported on a countable subset of $\mathbb{R}$. $\endgroup$ – J.Beck Jul 11 '18 at 20:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.