This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

1
vote
1answer
27 views

Question about the Shannon Entropy formula

I have a basic question about the Shannon Entropy formula. In fact it's so dumb that I didn't dare ask it in the class because I don't understand the text books. Here's the formula: $$H(X) = -\...
2
votes
0answers
36 views

Mathematical Definition of Entropy and a Question about the Nature of Stat. Mechanics Approach

I have been studying for quite some time now about entropic functionals, including Boltzmann-Gibbs, Renyi, Kaniadakis and Tsallis, and I am familiar with the properties that a functional has to ...
0
votes
0answers
16 views

Decomposition of Shannon conditional mutual information as seen on Wikipedia [Solved]

I am looking for a proof for a formula seen on Wikipedia: \begin{align} I(X;Y|Z) & = H(Z|X) + H(X) + H(Z|Y) + H(Y) - H(Z|X,Y) - H(X,Y) - H(Z) \\ {} & = I(X;Y) + H(Z|X) + H(Z|Y) - H(Z|X,Y) - H(...
1
vote
0answers
55 views

Difference of Entropy of two-dimensional Gaussians

I encountered a putative contradiction. Assume we have two 2-dim. Gaussian variables $z_1 = (x_1, y_1)$ and $z_2 = (x_2, y_2)$ with all components being independent, normal distributed variables: $x_1,...
0
votes
2answers
28 views

Derivation of joint entropy $H(X,Y) = H(X) + H(Y|X)$

Can somebody explain the calculations with arrows below? And I am sorry if I have placed my post in the wrong place.
0
votes
1answer
21 views

Information Channel Capacity

1) Suppose a Noiseless Binary Channel, who's input is reproduced exactly at the output. Let X be the transmitter and Y the receiver (i.e (X=0----->Y=0 and X=1-----> Y= 1)) I understand intuitively ...
4
votes
1answer
493 views

Notation of cross entropy

I have a question regarding a notation that seems to be very usual. For starters, cross entropy is defined by: \begin{align}H(X, q) &= H(X) + D(p||q) \\ & =-\sum_x p(x)\log_2 q(x)\end{align} ...
0
votes
0answers
66 views

Help me understand the proof for Shannon's Theorem 4 (regarding number of sequences of various probabilities) in the original paper

I'm reading Shannon's 1948 paper where I encountered Theorem 4: $$ \lim \limits_{N \to \infty} \frac {\log n(q)} N = H $$ In Appendix 3 after proving Theorem 3, Shannon proves Theorem 4 by saying: ...
0
votes
1answer
8 views

How to re-estimate a theoretical outcome knowing rng is imperfect

I have 3 regular dice with six sides. I am playing a game where I win when I match all three dice to the same side. My expected shannon bits of entropy H(X) evaluates to 7.754887502 bits. However, my ...
0
votes
0answers
34 views

mutual information and combinatorics

\begin{align} &\mathrm{H}\left(\frac{1}{2^{k}}\right) \\[3mm]&\ \!\!\!\!\!\!\!\!\!\! - {1 \over 2^{k}}\left\{% {k \choose 0}\mathrm{H}\left(\left[1 - \epsilon\right]^{\,k}\right) + {k \choose ...
5
votes
1answer
81 views

Is the Entropy a Function or a Functional? [duplicate]

As in the title, I was wondering whether the entropy of a system (it can be any entropy, from Boltzmann to Renyi etc, it is of no importance) is a function or a functional and why? Since it is mostly ...
1
vote
1answer
18 views

Is there a connection between topological entropy and stationary distributions?

In a book I read the following: "The topological entropy is the supremum over all stationary distributions of the entropy of the corresponding stationary sequence." I did not find this definition ...
2
votes
2answers
182 views

Formula for proportion of entropy

Let's say we have a probability distribution having 20 distinct outcomes. Then for that distribution the entropy is calculated is $2.5$ while the maximal possible entropy here is then of course $-\ln(\...
0
votes
1answer
20 views

Entropy, stirling's approximation

Given that the $\Omega$ is the total theoretical information encoded in a string of characrs let's say, we can express it in the bit-manner: $$\Omega=2^G$$ Therefore $G=\log_2 \Omega$ We know that $...
0
votes
0answers
17 views

Wave Equation: Distribution which maximises Entropy

Given the wave equation: $\left(\frac{\partial^2}{\partial t^2}-\frac{\partial^2}{\partial x^2}\right)f(t,x)=0$ and expanding $f(t,x)$ through a Fourier Transform: $f(t,x)=\int d\omega dk F(\omega,k)e^...
2
votes
1answer
47 views

why can't sort 12 elements in 29 comparisons

The information theoretic lower bound for sorting 12 elements is using 29 comparisons, but actually we can't sort them in less than 30 comparisons. My problem is that why we can't reach the ...
3
votes
2answers
100 views

Is conditional entropy ever taken to be a random variable?

In probability theory, the conditional expectation $E(X|Y)$ and variance $V(X|Y)$ er usually taken to be random variables, st. the value of $E(X|Y)$ depends on what value $Y$ ends up taking. I've ...
0
votes
1answer
10 views

Definition of Entropy for Homogeneous sample.

In the context of decision tree's we learned that Entropy of a Sample is defined as $H(<P_1...P_n>) = \sum_{i=1}^{n}P_i log_2(\frac{1}{P_i})$ Where $P_i$ is the proportion of the Variable i of ...
1
vote
1answer
36 views

Conceptual Question : Relationship between entropy and a technique for source coding

I want to encode the messages to a sequence of 1s and 0s (subsequently called "bits"). This is called "source coding". Shannon's source coding theory states that the entropy of a source that emits a ...
2
votes
3answers
49 views

How is the entropy of the normal distribution derived?

Wikipedia says the entropy of the normal distribution is $\frac{1}2 \ln(2\pi e\sigma^2)$ I could not find any proof for that, though. I found some proofs that show that the maximum entropy resembles ...
15
votes
4answers
5k views

Is there a “most random” state in Rubik's cube?

Is there a state in Rubik's cube which can be considered to have the highest degree of randomness (maximum entropy?) asssuming that the solved Rubik's cube has the lowest?
0
votes
1answer
26 views

Log base change problem, Multivariate Gaussian differential entropy proof

I am working through a proof in this document http://ee.tamu.edu/~georghiades/courses/ftp647/Chapter7.pdf for Theorem 3 (The entropy of a multivariate Gaussian distribution): Let X = (X1, X2, · · ·...
6
votes
3answers
956 views

What is information theoretic entropy and its physical significance?

I have learned entropy in my information theory classes. The definition I got from text books was the average information content in a message sequence etc. But in one of the MIT videos related to ...
1
vote
0answers
16 views

compute the L^2-distance of a given function to the set of Gaussian functions

I am faced with the following question: given a probability density function $f$ over $\mathbb{R}$ with $\int_{\mathbb{R}}f(x)x^2dx=\sigma^2$ given, I am trying to find the "nearest" Gaussian to $\...
0
votes
0answers
10 views

Nearly a factor map (only shifted)

For topological entropy, we have that $h(T)\geq h(S)$ if $(Y,S)$ is a topological factor of $(X,T)$, i.e. $T=\phi S\phi^{-1}$ where $\phi\colon X\to Y$ is a continuous surjection. That is we have $\...
1
vote
1answer
45 views

Rate distortion function with infinite distortion

I am working through the problems in Elements of Information Theory by Cover and Thomas and have come across the following problem I couldn't answer. The problem is to find the rate distortion ...
1
vote
1answer
23 views

Differential entropy of $\Gamma$

Let $X \sim Gamma(\alpha,\beta)$ be gamma distributed random variable with probability distribution function $$ f_{X}(x)=\frac{\beta^{\alpha}x^{\alpha-1}e^{-\beta x}}{\Gamma(\alpha)},\;x>0 $$ ...
5
votes
1answer
117 views

Looking for a a measure-theoretic treatment of “differential entropy”

If $X$ is a discrete random variable, its entropy $H(X)$ is usually defined as something along the lines of $-\sum \def\P{\mathbb{P}}\P(x) \log_2( \P(x))$, where the sum ranges over all the possible ...
13
votes
5answers
409 views

(Elegant) proof of an inequality: $h(x) \geq 1- (1-\frac{x}{1-x})^2$, where $h$ is the binary entropy function

I am looking for the most concise and elegant proof of the following inequality: $$ h(x) \geq 1- \left(1-\frac{x}{1-x}\right)^2, \qquad \forall x\in(0,1) $$ where $h(x) = x \log_2\frac{1}{x}+(1-x) \...
0
votes
1answer
32 views

Comparing entropies $H((f(X,Y), g(X,Y)))$ and $H ((f(X,Y),g(X,Z)))$

Let X,Y,Z be three independent uniform distributions on $\{0,1\}^n$; $f, g:\{0,1\}^n\times\{0,1\}^n\rightarrow\{0,1\}$ be two boolean functions. Is it true that $$H((f(X,Y), g(X,Y)))\leq H ((f(X,Y),...
1
vote
0answers
33 views

Conditional entropy $H(A,B|C)$ independent of whether $p(A,B)$ factorises or not

I arrived at a (slightly different but hopefully more precise) reformulation of my original question shown below: Is $H(A,B)-H(A,B|C)$ maximal for independent $A$ and $B$ in a similar sense as $H(A,B)...
-2
votes
1answer
39 views

Does, S = k ln W == W = e^s/k? [closed]

"Boltzmann's equation relates the entropy S of an ideal gas to the number W of microstates corresponding to a given macrostate, via the equation S = k ln W where k is the so-called Boltzmann ...
0
votes
0answers
14 views

Anti-intuition when finding statistical model of a random variable using Maximum Entropy Principal

I was trying to understand the Maximum Entropy Principal, and was calculating a very simple example, but ran into some confusion. Consider a random variable $X$, which can only take values $1,2$ and $...
1
vote
0answers
24 views

Calculating change in entropy, Integration help?

So I need to calculate the change in entropy for a non-ideal gas over two states. The equation im looking at using is $$T\,ds = dh - v\,dP \text{ or } T\,ds = du + P\,dv$$ where $h=u+Pv$ , $s = \...
1
vote
3answers
91 views

Partition-based entropy of a sequence

The entropy $H$ of a discrete random variable $X$ is defined by $$H(X)=E[I(X)]=\sum_xP(x)I(x)=\sum_xP(x)\log P(x)^{-1}$$ where $x$ are the possible values of $X$, $P(x)$ is the probability of $x$, $...
1
vote
1answer
51 views

Is there a statistical measure of bitwise entropy?

(Somewhat inspired by this website, particularly Section III. Also, I might be using a different definition of entropy than usual; what I am using is closest to the physics definition (the one I ...
2
votes
0answers
37 views

How to evaluate the quality of the probability distribution output of a classifier?

In a classification problem, I have trained a neural network which outputs class probabilities for a given input. For a new input, I now want to evaluate the "quality" of the neural network's ...
0
votes
0answers
25 views

Change of bases for entropy

From Cover and Thomas, Elements of Information Theory: Why isn't it: $ \log_b(p) = \frac{\log_a(p)}{\log_a(b)} $, so that $ H_a(X) $ is multiplied with $ \frac{1}{\log_b a} $?
1
vote
0answers
22 views

Entropy of a 2-dimensional function versus 1-dimensionl function.

I am a novice in information theory so this is more of a question seeking pointers to ideas/references to think further on the thought. I want to make concrete the idea that a function of two ...
-1
votes
1answer
21 views

Does the Information Gain algorithm favor a high-entropy attribute or a low-entropy one?

This might not be mutual to mathematics but it does relate to Information-Theory. My question is: Does the InformationGain algorithm, in Decision-Tree machine-learning, favor a high-entropy ...
1
vote
1answer
15 views

Simplyfying Shannon Entropy formula

I need to calculate the following which is Shannon Entropy formula only using simple functions like log or squareroot or I don't know, just make it simple enough for a guy that does understand only ...
2
votes
1answer
97 views

Random permutations composition

I'm trying to prove a theorem that seems very intuitive. However, I seem to be missing a piece of the puzzle. If: $\pi$ is a random permutation ($S_n$), $\pi_1, \pi_2$ - random permutations with ...
6
votes
2answers
2k views

Derivation of the maximum entropy distribution

I am reading a book and having trouble following something. The problem is to try to maximize the differential entropy $-\int_{0}^{\infty}p(r)\log p(r)$ under the constraints that $\int_{0}^{\infty}p(...
0
votes
1answer
58 views

Conditional entropy under quantization

Let $X$ be a continuous random variable and $X^n$ its quantization that becomes finer with larger $n$. Let $Y$ be a deterministic function of $X$. Then we have that the conditional entropy $$H(Y|X) = ...
2
votes
1answer
103 views

proof of upper bound on differential entropy of f(X)

I asked a similar question yesterday, but I organized my question here a little and further asked my second question. Suppose $X$ is a continuous random variable with the pdf $f_x$, and $Y=g(X)$. If ...
1
vote
0answers
25 views

How to test quality of probability estimates?

I have a Markov chain model which produces a probability distribution for absorption in 4 possible absorbing states. I.e. the model estimates the probability distribution for a discrete random ...
1
vote
0answers
17 views

Channel capacity of sum of symmetric channels

I've got a channel matrix $P$ of the form $\begin{bmatrix} Q \\ R \end{bmatrix}$ where $Q,R$ are channel matrices of symmetric channels, so they now have different input alphabets but the ...
0
votes
1answer
28 views

Basic Entropy Inequality and Identity question

This is a solution to a problem I am working on: \begin{equation} \begin{aligned} H(X|Y) + H(Y|Z) &\ge^? H(X|Y, Z) + H(Y|Z) \\ &=^\text{?}H(X,Y |Z) \\ &= H(X|Z) + H(Y|X, Z)\\ &\ge H(X|...
0
votes
0answers
21 views

Entropy of degenerate multivariate normal

After having read the Wikipedia entry on the degenerate case of the multivariate normal distribution: https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case my question is: ...
0
votes
0answers
51 views

How to calculate the Shannon Entropy for a block length of a word

I have a binary sequence of length N as $10110110111...$ I want to segment the above series into equal blocks of a window of length $L$. One way of determining the block length is using the ...