How do define the entropy of a probability measure? What is the motivation to define it? What is the importance of a probability measure to minimize or maximize the entropy? In a finite sample space is there a combinatorial interpretation?

I thank you for the answers.

Edit1. @Louis for the comment, I can improve my question. Now would be pleased to know that the sufficient conditions on the probability measure (or on its density) to maximize the Shannon entropy. I am mainly interested in the case of multinomial probability distribution.

Edit2. The Shannon entropy of $P_X(\cdot)=P(X\in \cdot)$ is defined to $S(P_X)=\int_{\Omega}-X\cdot ln(X) d P(X)$. Here, X is a random variable on the probability space $(\Omega, \mathcal{F}, P)$ and $-X\cdot ln(X)=0$ if $X\leq 0$.

  • 1
    $\begingroup$ Maybe this wikipedia link can help en.wikipedia.org/wiki/Entropy_(information_theory) $\endgroup$ – Louis Feb 20 '12 at 12:04
  • $\begingroup$ @ Louis, Thanks for your link. Helped to refine my question. I'm enteressado exactly in the Shannon entropy. $\endgroup$ – MathOverview Feb 20 '12 at 12:37
  • 1
    $\begingroup$ I recommend Billingsley's Ergodic Theory and Information for an exposition of these ideas. $\endgroup$ – Quinn Culver Feb 20 '12 at 13:37

Entropy $H=-\sum_i p_i\log p_i$ in the discrete case and $H=-\int f(x)\log f(x)\,dx$ (where $f$ is the probability density function) in the continuous. The choice of base of logarithm is free, but affects $H$ as a scale factor. It represents the amount of uncertainty in a stochastic system or model, or the combinatorial complexity of a finite sample space, i.e. how hard it should be a priori to compress the contained information, without any prior shared reference point.

Claude Shannon's motivation was to formulate and quantify the amount of information contained (stored, transmitted, increased or lost), in a statistical sense, within various forms of communication. He is widely credited as the father of information theory, which also has deep connections to thermodynamis, data compression, where asymptotic equipartition property the plays a fundamental role, and even bayesian statistics, where the principle of maximum entropy becomes an axiom. The concept of entropy is in fact so fundamental that one can also talk of conditional entropy, joint entropy, just as in statistics, or relative entropy.

Chris Hillman states in his An Entropy Primer that Shannon's Noiseless Coding Theorem and a general version of the Equipartition Theorem, together, give Shannon's way of understanding what his entropy "means". But in What is Information, he warns:

There are also some non-relations which are insufficiently appreciated. For instance, the "entropy" of a density is analogous not to probabilistic entropy but rather to a related quantity called divergence.

Or as Mohri, et. al. 2008 say, citing the same work:

The relative entropy, or Kullback-Leibler divergence, is one of the most commonly used measures of the discrepancy of two distributions p and q [Cover and Thomas, 1991].

And from wikipedia:

Another idea, championed by Edwin T. Jaynes, is to use the principle of maximum entropy (MAXENT). The motivation is that the Shannon entropy of a probability distribution measures the amount of information contained in the distribution. The larger the entropy, the less information is provided by the distribution. Thus, by maximizing the entropy over a suitable set of probability distributions on X, one finds the distribution that is least informative in the sense that it contains the least amount of information consistent with the constraints that define the set. For example, the maximum entropy prior on a discrete space, given only that the probability is normalized to 1, is the prior that assigns equal probability to each state. And in the continuous case, the maximum entropy prior given that the density is normalized with mean zero and variance unity is the standard normal distribution. The principle of minimum cross-entropy generalizes MAXENT to the case of "updating" an arbitrary prior distribution with suitability constraints in the maximum-entropy sense.


Andrej N. Kolmogorov defined the entropy of a metric space $(X,d)$ as a function (!) on natural numbers $n$: the logarithm of the least number of balls of $d$-diameter $2^{-n}$ sufficient to cover $X$. The discrete space $\{1,...,2^k\}$ has constant entropy $k$, coinciding with the Shannon Entropy of the uniform distribution on it. This suggests the following synthesis:

Let the entropy of a Borel measure $\mu$ on a totally bounded metric space $(X,d)$ be the function assigning to each natural number $n$ the infimum of $-\sum_j \mu(X_j)\cdot\log\mu(X_j)$, taken over all covers of $X$ by Borel subsets $X_1,...,X_J$ of $d$-diameter $2^{-n}$.

For a discrete space this boils down to Shannon's notion; and Kolmogorov's is recovered for $[0;1]$ equipped with the uniform distribution. (It may or may not be related to the limiting density...)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.