0
$\begingroup$

This question is NOT a duplicate of this. Please do not close this as a duplicate. The problem is similar but I am not satisfied with the solution provided.

According to deep learning textbook by Ian Goodfellow (with regards to information theory):

we define the self-information of an event $x=x$ to be $ I(x)=−log(P(x)) $. When $x$ is continuous, we use the same definition of information by analogy, but some of the properties from the discrete case are lost. For example, an event with unit density still has zero information, despite not being an event that is guaranteed to occur.

To find the probability of such an event we find the cumulative density function of unit probability density function.

In the answer to this question it was stated that

[...] The cumulative density function of such a PDF is zero

Implying that the cumulative density function of the unit probability density function is $0$. I don't understand how this can be true.

If $D(x)$ is comulative density of function and P(x) is probablity density function then

$ D(x) = \int P(x) \,dx $

$ D(x) = \int 1 \,dx $ (unit probability density function means $ P(x) = 1 $)

$ D(x) = x + C $

Which is clearly not equal to zero, well unless limits are equal. Is don't see why limits would always be equal here. If the limits are equal probability of any continuous variable is zero, which is not unique to the unit probability density function. Then why would the author even mention the unit?

So my question is why cumulative density function of a unit probability density function zero?

$\endgroup$

1 Answer 1

1
$\begingroup$

From what I've seen so far of this book, I'm not a big fan. The author plays very fast and loose with the notions of a probability measure and a probability density/mass function. The distinction is not so important for discrete RVs, but extremely important for continuous RVs.

For example, suppose $\mathrm x\sim \operatorname{Normal}(0,1)$. The probability density function for $\mathrm x$ is $$p_{\mathrm x}(x)=\frac{1}{\sqrt{2\pi}}\exp(-x^2/2)$$ So, e.g $$p_{\mathrm x}(1)=\frac{1}{\sqrt{2\pi}}\exp(-1/2)\approx 0.24197$$ The cumulative distribution is $$P_{\mathrm x}(x)=\int_{-\infty}^x p_\mathrm x(s)\mathrm ds=\frac{1}{2}\big(1+\operatorname{erf}(x/\sqrt{2})\big)$$ So $$P_{\mathrm x}(1)=\frac{1}{2}+\frac{1}{2}\operatorname{erf}(1/\sqrt 2)\approx 0.84134$$

But, the probability that $\mathrm x$ is equal to one is $$\Pr(\mathrm x=1)=\lim_{\epsilon\to 0}\int_{1-\epsilon}^{1+\epsilon}p_{\mathrm x}(x)\mathrm dx=0$$

In short, we have $$p_{\mathrm x}(1)=0.24197 \\ P_{\mathrm x}(1)=0.84134 \\ \Pr(\mathrm x=1)=0$$

However, in the ambiguous notation of the author, the last two lines would read $$P(1)=0.84134 \\ P(1)=0$$

This is likely the source of your confusion.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .