# Tagged Questions

0answers
46 views

1answer
54 views

### Using binary entropy function to approximate log(N choose K)

I am not a mathematician and struggling with the exercises while reading this book Information Theory, Inference and Learning Algorithms. The author introduced the binary entropy function at the ...
3answers
201 views

### Similarity between two probability distribution

I am not sure how to put the question. I am not even sure if this question makes sense at all. I know that the similarity of two discrete (or continuous) distributions can be quantified by ...
1answer
35 views

### A question on Markov chain

Suppose for two random variables $X$ and $Y$ we have $X\perp\!\!\!\perp Y$ and also assume that three random variables $X$, $Y$ and $Z$ form the following Markov chain: $X\to Z\to Y$. Do these two ...
0answers
60 views

### Doubts in Bayes' Theorem

I meet one problem on the probability and statistic theory. "Assume given the probability spaces $(X,S,\mu_i)$, $i=1,2$, and the probability space $(X,S,\lambda)$. And there exsit functions ...
2answers
43 views

### Does X|Y = X formally, in the sense of RVs?

In Cover and Thomas' "Elements of Information Theory", the joint entropy $H(X,Y)$ is defined, but they state that this definition is nothing new if we consider that it is the entropy of a single ...
2answers
82 views

### Bounding second moment of entropy

Entropy is defined as $E(-\log(P(x))$. We know it is bounded by $\log(r)$ when $r$ is the size of alphabet. Defining the second moment as $E(\log^2(P(x))$, how to show it is bounded?
3answers
59 views

### Why can we use entropy to measure the quality of a language model?

I am reading the < Foundations of Statistical Natural Language Processing >. It has the following statement about the relationship between information entropy and language model: ...The ...
1answer
54 views

### Shannon Entropy Minimization

The Shannon Entropy for an observation is given by $-x \log_2(x)$. Why is the maximum entropy achieved at $x = \frac{1}{e}$, and not at $x = 0$? Could someone provide a logical explanation that ...
1answer
55 views

### Jensen's inequality for countable probability space

One form of Jensen's inequality for the finite case, tells us that $$\sum_{x \in X} p(x) \log q(x) \leq \log\sum_{x \in X} p(x) \cdot q(x)$$ For positive p(x), and $\sum_{x \in X} p(x) = 1$, ...
2answers
88 views