0
votes
1answer
50 views

How does the presenter in this video derive this formula?

I am watching this coursera video on entropy (in the information theory sense of the word). Right around the two minute mark the presenter shows two forms for H(p). The first (after the equals sign) ...
1
vote
0answers
181 views

What do two number on top of each other in square brackets mean?

Im currently going through "Universal Portfolios with Side Information" by Cover and Ordentlich [96]. Near the end of the paper, they provide a formula for calculating weights of a Universal Portfolio ...
2
votes
2answers
107 views

Understanding notation difference between mutual information and information divergance

The mutual information is defined on random variables. That is, $I(X;Y)$ denotes the mutual information between random variables $X$ and $Y$. On the other hand, the the Kullback-Leibler divergence is ...
0
votes
1answer
47 views

What is being maximised in the channel capacity formula?

The channel capacity formula is given as such: $$C=\max_{p(x)}I(X,Y)$$ Does this mean that it is the maximum probability multiplied by the mutual information, or is something else being maximised ...
8
votes
2answers
727 views

Why is “h” used for entropy?

Why is the letter "h" (or "H") used to denote entropy in information theory, ergodic theory, and physics (and possibly other places)? Edit: I'm looking for an explanation of the original use of "H". ...
1
vote
1answer
151 views

Cover & Thomas textbook notation

Exercise 2.27 in Elements of Information Theory (2nd ed.) reads: Let $\mathbf{p} = (p_{1}, p_{2}, \ldots, p_{m})$ be a probability distribution on $m$ elements (i.e., $p_{i} \geq 0$ and ...