Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm reading a paper in which it is stated that

The entropy of an ergodic measure is defined as $$\lim_{n \to \infty} -\frac{1}{n} \sum_{|w|=n} \mu[w] \log \mu[w].\tag{1} \label{eq:1}$$

Here the underlying space is $(\{0,1\}^{\mathbb{N}}, \mathscr{B}, \mu)$, where $\mathscr{B}$ is the $\sigma$-algebra generated by cylinders. So for a fixed $n$, the sum is over all binary strings of lenght $n$.

I've not seen the entropy of a measure defined anywhere, so I need to make sure I know what that means. Since the definition at (1) above is actually of the entropy of an ergodic measure, and a measure can only be ergodic with respect to some underlying measure-preserving transformation, it seems that the transformation must be the shift since the shift, $T$, is already mentioned. With that assumption, I conclude then, that the entropy of $\mu$ must be what most books (that I've seen) call the entropy of $T$.

Question 1: Is my interpretation of the definition at (1) correct?

If the answer to question 1 is yes, then the definition of the entropy of $T$ (or $\mu$) differs from the one I've seen, which is that $$h(T) = \sup_{\mathcal{A}} \, h(T,\mathcal{A}),\tag{2} \label{eq:2}$$ where $\mathcal{A}$ ranges over all finite (measurable) partitions and for any fixed finite partition $\mathcal{A}$, $$h(T,\mathcal{A}) = \limsup_{n\to \infty} \frac{1}{n} H \left( \bigvee_{k=0}^{n-1} T^{-k} \mathcal{A} \right),$$ where $\bigvee$ denotes the common refinement of partition (i.e. the join).

Question 2: Are the definitions at (1) and (2) equivalent in this scenario?

share|cite|improve this question
Apparently the \eqref command doesn't work here, so my references to equations (1) and (2) are hard-coded. If someone knows how to fix this, please do (and/or inform me). – Quinn Culver Nov 27 '11 at 16:54
Assuming the answer to question 1 is "yes", I think the answer to question 2 is also "yes" because of the Kolmogorov-Sinai theorem. – Quinn Culver Nov 27 '11 at 17:05
up vote 2 down vote accepted
  • Consider the context of your second question. On the underlying probability space, let ${(Z_n)}_{n \geq 0}$ be the stationary (shift-invariant) process defined by $Z_n(x)=P(T^n(x))$ where $P(x)$ denotes the element of the partition $P$ to which $x$ belongs. Then $h(T,P)=\lim \frac{H(Z_1,\ldots,Z_n)}{n}$ where $H(\cdot)$ denotes the entropy of a discrete random variable.

  • Now consider the context of your first question. As you feel, the transformation $T$ coming into play is the shift. The partition $P$ of $(\{0,1\}^{\mathbb{N}}$ defined by the equivalence relation $(x_n) \sim (y_n)$ $\iff$ $x_0=y_0$ is a generator of $T$ and then Kolmogorov & Sinai's theorem mentioned by yourself says that $h(T)=h(T,P)$.

That shows that "yes" is the answer to your both questions, because here $H(Z_1,\ldots,Z_n)= - \sum_{|w|=n} \mu[w] \log \mu[w]$.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.