# Tagged Questions

This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

27 views

33 views

24 views

51 views

### Is there a statistical measure of bitwise entropy?

(Somewhat inspired by this website, particularly Section III. Also, I might be using a different definition of entropy than usual; what I am using is closest to the physics definition (the one I ...
37 views

### How to evaluate the quality of the probability distribution output of a classifier?

In a classification problem, I have trained a neural network which outputs class probabilities for a given input. For a new input, I now want to evaluate the "quality" of the neural network's ...
25 views

### Change of bases for entropy

From Cover and Thomas, Elements of Information Theory: Why isn't it: $\log_b(p) = \frac{\log_a(p)}{\log_a(b)}$, so that $H_a(X)$ is multiplied with $\frac{1}{\log_b a}$?
22 views

### Entropy of a 2-dimensional function versus 1-dimensionl function.

I am a novice in information theory so this is more of a question seeking pointers to ideas/references to think further on the thought. I want to make concrete the idea that a function of two ...
21 views

### Does the Information Gain algorithm favor a high-entropy attribute or a low-entropy one?

This might not be mutual to mathematics but it does relate to Information-Theory. My question is: Does the InformationGain algorithm, in Decision-Tree machine-learning, favor a high-entropy ...
15 views

### Simplyfying Shannon Entropy formula

I need to calculate the following which is Shannon Entropy formula only using simple functions like log or squareroot or I don't know, just make it simple enough for a guy that does understand only ...
I'm trying to prove a theorem that seems very intuitive. However, I seem to be missing a piece of the puzzle. If: $\pi$ is a random permutation ($S_n$), $\pi_1, \pi_2$ - random permutations with ...