Shannon information is defines as, \begin{equation} H(p_1, p_2,...p_n) = - \sum_{i=1}^{N} p_i log p_i.\end{equation} For continues distribution, we can write shanon information as, \begin{equation}H(x) = \int^{\infty}_{-\infty} p(x) log p(x).\end{equation} For equation one, it is for discrete case. It is showing shanon entropy for set of N outcomes. I want to ask that for continues distribution, we are taking integration limit to infity. what does it means does it means that we are considering infinite number of events? can we take integration from 0 to 1 or 0 to 2?


For continuous probability distributions, we have what is called differential entropy. For a continuous random variable X with probability density function p(x) defined on the interval I, the entropy is given by the expression as you have given, $$H(X) = -\int_{I}p(x)log(p(x))dx$$ For example, Let's say we have a normal probability distribution given by, $$p(x) = \frac{1}{\sqrt{2\pi}}e^{-\frac{x^2}{2}}$$ This function has a non-zero value everywhere in the interval I = ($-\infty,\infty$) and tends to zero at infinity and hence the limits of integration. Looks something like this,

enter image description here



Discrete and Continuous probability distributions

Note a discrete random variable https://en.wikipedia.org/wiki/Random_variable $X$ takes on discrete values say for example in $\mathbb{N}$ with positive probability. Discrete random variables can be described by their probability mass function $p$. These describe experiments like dice rolling etc.

Continuous random variables are described by their probability density functions, and give distributions like the height of a population etc.


From an information theory point of view we can ask how much information we receive when we observe a specific value for a random variable $X$, if we are told a highly improbable event has just occurred we will receive more information than if we are told a very likely event has just occurred ( make sure you understand this sentance because it is at the heart of the description of entropy, this link will help : https://www.quora.com/Why-does-an-unlikely-event-give-us-more-information-than-a-likely-event#:~:text=The%20higher%20the%20probability%20of,previous%20beliefs%20will%20be%20disrupted. )

$\underline{\textbf{Constructing a function that measures the information content :}}$ Lets call this function $h(\cdot)$. Our measure of information content will be a monotonically increasing function of the probability mass function $p(x)$. Construction of such a function can be motivated by the following observation, that is, if we have two events that are unrelated then the information we gain from being both of them occur should be the same as the sum of the information that each of them occur separately :


Note two unrelated events will be independent, i.e


Using these two facts, one can show that $h$ must be given by the logarithm of $p$, so that

$$h(x)=-\log p(x)$$

  • Here we can (arbitrarily) choose to use the natural log, which has many nice links to physics. In information theory the convention is to use base two, whereby the units of $h(x)$ are 'bits'.
  • The negative sign ensures the entropy is positive or zero.

$\underline{\textbf{The entropy of a random variable : }}$ The average amount of information it would take for a sender to transmit value of a random variable to a receiver would then be the average of $h$ with respect to the probability distribution $p$ https://en.wikipedia.org/wiki/Expected_value, i.e :

$$ E[p]=-\sum_{x}p(x)\log(p(x)).~~~~~~(1) $$ This is called the entropy of the random variable $X$. I like to think of it as a $\textit{functional on the space of probability measures }$. Note the reason it is a sum is because we have assumed $p$ is a mass function associated to a discrete random variable. Then a natural way to extend the definition of entropy to continuous random variables is to define it as the expectation of $\log(p(X))$ for a continuous random variable $X$ giving

$$ E[p]=-\int p(x)\log p(x).~~~~~~(2) $$

To be properly defined note that in $(1)$ as a functional the entropy is defined as a map acting on the space of probability measures over some discrete set whereas in $(2)$ it is defined over probability measures on a continuous set - Infact in $(2)$ we have assumed that the measure of $X$ admits a density which we have called $p$.

Your question : $"\textbf{can we take the integration from 0 to 1 ?}$" Answer - no you must integrate over the support of the distribution of your random variable, by the above construction it would not make sense to do otherwise.

There is a more in-depth discussion of entropy from a information theory and statistical mechanic point of views in the book "Pattern Recognition and Machine Learning" of Bishop - as well as a detailed description of going from the discrete to the continuum.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.