This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

-1
votes
0answers
32 views

Entropy and Markov chain [on hold]

Assume that $X_n$ is a discrete Markov chain and $H$ is entropy function. I want to prove $$H\left(X_0\mid X_n\right) \geq H\left(X_0\mid X_{n-1}\right)$$ but I have no idea how to prove it. please ...
1
vote
2answers
81 views
+50

Formula for proportion of entropy

Let's say we have a probability distribution having 20 distinct outcomes. Then for that distribution the entropy is calculated is $2.5$ while the maximal possible entropy here is then of course $-\ln(\...
-1
votes
0answers
93 views

I need help in Entropy question (information Theory)

I need help in proving an inequality: $1 - H(p,q) \leq |p-q|$ and $H(p,q) \geq 2p$ and $H(p,q)\geq 2 \min(p,q)$? Thanks in advanced!
0
votes
1answer
20 views

Entropy, stirling's approximation

Given that the $\Omega$ is the total theoretical information encoded in a string of characrs let's say, we can express it in the bit-manner: $$\Omega=2^G$$ Therefore $G=\log_2 \Omega$ We know that $...
0
votes
0answers
15 views

Wave Equation: Distribution which maximises Entropy

Given the wave equation: $\left(\frac{\partial^2}{\partial t^2}-\frac{\partial^2}{\partial x^2}\right)f(t,x)=0$ and expanding $f(t,x)$ through a Fourier Transform: $f(t,x)=\int d\omega dk F(\omega,k)e^...
0
votes
1answer
10 views

Definition of Entropy for Homogeneous sample.

In the context of decision tree's we learned that Entropy of a Sample is defined as $H(<P_1...P_n>) = \sum_{i=1}^{n}P_i log_2(\frac{1}{P_i})$ Where $P_i$ is the proportion of the Variable i of ...
2
votes
1answer
45 views

why can't sort 12 elements in 29 comparisons

The information theoretic lower bound for sorting 12 elements is using 29 comparisons, but actually we can't sort them in less than 30 comparisons. My problem is that why we can't reach the ...
1
vote
1answer
35 views

Conceptual Question : Relationship between entropy and a technique for source coding

I want to encode the messages to a sequence of 1s and 0s (subsequently called "bits"). This is called "source coding". Shannon's source coding theory states that the entropy of a source that emits a ...
2
votes
3answers
43 views

How is the entropy of the normal distribution derived?

Wikipedia says the entropy of the normal distribution is $\frac{1}2 \ln(2\pi e\sigma^2)$ I could not find any proof for that, though. I found some proofs that show that the maximum entropy resembles ...
1
vote
0answers
16 views

compute the L^2-distance of a given function to the set of Gaussian functions

I am faced with the following question: given a probability density function $f$ over $\mathbb{R}$ with $\int_{\mathbb{R}}f(x)x^2dx=\sigma^2$ given, I am trying to find the "nearest" Gaussian to $\...
0
votes
0answers
10 views

Nearly a factor map (only shifted)

For topological entropy, we have that $h(T)\geq h(S)$ if $(Y,S)$ is a topological factor of $(X,T)$, i.e. $T=\phi S\phi^{-1}$ where $\phi\colon X\to Y$ is a continuous surjection. That is we have $\...
1
vote
1answer
23 views

Differential entropy of $\Gamma$

Let $X \sim Gamma(\alpha,\beta)$ be gamma distributed random variable with probability distribution function $$ f_{X}(x)=\frac{\beta^{\alpha}x^{\alpha-1}e^{-\beta x}}{\Gamma(\alpha)},\;x>0 $$ ...
0
votes
1answer
32 views

Comparing entropies $H((f(X,Y), g(X,Y)))$ and $H ((f(X,Y),g(X,Z)))$

Let X,Y,Z be three independent uniform distributions on $\{0,1\}^n$; $f, g:\{0,1\}^n\times\{0,1\}^n\rightarrow\{0,1\}$ be two boolean functions. Is it true that $$H((f(X,Y), g(X,Y)))\leq H ((f(X,Y),...
1
vote
0answers
32 views

Conditional entropy $H(A,B|C)$ independent of whether $p(A,B)$ factorises or not

I arrived at a (slightly different but hopefully more precise) reformulation of my original question shown below: Is $H(A,B)-H(A,B|C)$ maximal for independent $A$ and $B$ in a similar sense as $H(A,B)...
-2
votes
1answer
39 views

Does, S = k ln W == W = e^s/k? [closed]

"Boltzmann's equation relates the entropy S of an ideal gas to the number W of microstates corresponding to a given macrostate, via the equation S = k ln W where k is the so-called Boltzmann ...
0
votes
0answers
13 views

Anti-intuition when finding statistical model of a random variable using Maximum Entropy Principal

I was trying to understand the Maximum Entropy Principal, and was calculating a very simple example, but ran into some confusion. Consider a random variable $X$, which can only take values $1,2$ and $...
1
vote
0answers
21 views

Calculating change in entropy, Integration help?

So I need to calculate the change in entropy for a non-ideal gas over two states. The equation im looking at using is $$T\,ds = dh - v\,dP \text{ or } T\,ds = du + P\,dv$$ where $h=u+Pv$ , $s = \...
1
vote
1answer
14 views

Simplyfying Shannon Entropy formula

I need to calculate the following which is Shannon Entropy formula only using simple functions like log or squareroot or I don't know, just make it simple enough for a guy that does understand only ...
13
votes
5answers
404 views

(Elegant) proof of an inequality: $h(x) \geq 1- (1-\frac{x}{1-x})^2$, where $h$ is the binary entropy function

I am looking for the most concise and elegant proof of the following inequality: $$ h(x) \geq 1- \left(1-\frac{x}{1-x}\right)^2, \qquad \forall x\in(0,1) $$ where $h(x) = x \log_2\frac{1}{x}+(1-x) \...
1
vote
0answers
25 views

How to test quality of probability estimates?

I have a Markov chain model which produces a probability distribution for absorption in 4 possible absorbing states. I.e. the model estimates the probability distribution for a discrete random ...
1
vote
0answers
15 views

Channel capacity of sum of symmetric channels

I've got a channel matrix $P$ of the form $\begin{bmatrix} Q \\ R \end{bmatrix}$ where $Q,R$ are channel matrices of symmetric channels, so they now have different input alphabets but the ...
0
votes
1answer
26 views

Basic Entropy Inequality and Identity question

This is a solution to a problem I am working on: \begin{equation} \begin{aligned} H(X|Y) + H(Y|Z) &\ge^? H(X|Y, Z) + H(Y|Z) \\ &=^\text{?}H(X,Y |Z) \\ &= H(X|Z) + H(Y|X, Z)\\ &\ge H(X|...
0
votes
0answers
13 views

Entropy of degenerate multivariate normal

After having read the Wikipedia entry on the degenerate case of the multivariate normal distribution: https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case my question is: ...
0
votes
1answer
21 views

Calculating Entropy and Information Gain of a Variable

I have the following values for two random variables. I need to compute the following values: a. H(Y) b. H(Y|X) c. and finally IG(Y|X) I will show what I have calculated so far. a. H(Y) = -(.5*...
0
votes
1answer
17 views

Estimation for entropy

Let $T\colon X\to X$ be continuous and $X$ compact and $K\subset X$ compact. By $s_n(2^{-k},K,T)$ denote the maximal cardinality of any $(n,2^{-k})$ separated subset of $K$. Suppose, we know for ...
2
votes
1answer
95 views

proof of upper bound on differential entropy of f(X)

I asked a similar question yesterday, but I organized my question here a little and further asked my second question. Suppose $X$ is a continuous random variable with the pdf $f_x$, and $Y=g(X)$. If ...
1
vote
1answer
49 views

Is there a statistical measure of bitwise entropy?

(Somewhat inspired by this website, particularly Section III. Also, I might be using a different definition of entropy than usual; what I am using is closest to the physics definition (the one I ...
0
votes
0answers
40 views

How to calculate the Shannon Entropy for a block length of a word

I have a binary sequence of length N as $10110110111...$ I want to segment the above series into equal blocks of a window of length $L$. One way of determining the block length is using the ...
0
votes
0answers
13 views

pattern-sensible entropy measure

I have some binary images (meaning each pixel can be 0 or 1), I want to find a pattern-sensible entropy measure, which means for example that a chessboard should have a very low entropy value (almost ...
0
votes
0answers
9 views

Can the entropy of a system of multiple stochastic processes be defined?

I'm trying to measure the entropy of BINGO cards. (waiting for laughter to die down.) Normally one would think of this as 75 choose 24, but in reality it is 15 choose 5, four times, and 15 choose 4, ...
0
votes
0answers
4 views

Calculating the entropy of original set E(S)

I am currently revising for an exam and I am struggling with this question Using a decision tree algorithm with information gain splitting, which of the two attributes diarrhea or fever is a ...
0
votes
0answers
11 views

Compare two distributions with varying focus on different regions

I have been trying to find if my problem matches has been discussed in prior research and if any technique exists to solve it. Here's the problem: Given two distributions (pdf) D1 and D2 over a ...
1
vote
1answer
16 views

Entropy of factor map

Let $A$ and $B$ be two compact metric spaces with $B\subset A$- Moreover, let $T\colon A\to A$ continuous and let $S\colon A\to B$ a continuous surjection with $S\circ T=T\circ S$. Moreover assume ...
2
votes
1answer
95 views

Random permutations composition

I'm trying to prove a theorem that seems very intuitive. However, I seem to be missing a piece of the puzzle. If: $\pi$ is a random permutation ($S_n$), $\pi_1, \pi_2$ - random permutations with ...
0
votes
0answers
13 views

Calculating Split Info with given equation not matching solution

I'm given the formula to calculate the Split info but cannot seem to calculate the correct answer of 0.926 that the example shows. Split Info $= -\Sigma \frac{\mid D_j\mid}{\mid D\mid} * log_2 (\frac{...
0
votes
1answer
22 views

Decomposition of the entropy

So, I'm reading about this property in the MacKay book. But I don't fully get it. Can someone explain it to me? There's this example: A source produces a character $x$ from the alphabet $A = \{0, ...
3
votes
2answers
64 views

Deriving the q-Gaussian PDF

Ok it may sound a bit too simple but I am quite confused here. While studying generalized entropic forms, in my case that of $S_q$ or in another words the Tsallis Entropy, I reach a point where I have ...
0
votes
1answer
20 views

Calculating mutual information for a dataset

I have a dataset of individual text documents $D = {d_0, d_1, ..., d_n}$ and a corpus of keywords $K = {k_0, k_1, ..., k_m}$ in the documents. There are zero or more keywords in each text document. I ...
1
vote
1answer
44 views

Conditional Entropy and Gibbs Inequality

We know $$H(X | Y) + H(Y) = H(X, Y)$$ Therefore, $$H(X | Y) \leq H(X, Y) $$ since $$ H(Y) \geq 0$$ If we expand this out, we get $$-\sum_{x,y} {p(x,y) \log p(x | y)} \leq - \sum_{x,y} {p(x,y) \log p(x,...
0
votes
1answer
36 views

Entropy/Variance inequality

The following inequality is sometimes used as a building block to prove log Sobolev inequalities. Does anyone have a simple proof of it? $$ x\log x + y\log y - (x+y)\log \frac{x+y}{2}\leq (\sqrt x-\...
1
vote
1answer
37 views

Convert a joint entropy matrix to a contidional entropy matrix.

I've only barely started to learn about Entropy and Information Theory as a part of a course I'm taking in Systems Theory / Cybernetics. The thing is, I'm terrible at math! Say I have a joint ...
2
votes
1answer
61 views

Password entropy of famous xkcd comic

The famous xkcd comic about password strength calculates the entropy of the 11-character password "Tr0ub4dor&3" with 28 bits of entropy. When following the ASCII-95-chart, we have 95 possible ...
1
vote
1answer
34 views

Does entropy inequality hold for convex combination

I have two pairs of Random Variables, $(\mathbb{X},\mathbb{Y})$ and $(\mathbb{M},\mathbb{N})$ which satisfies, $H(\mathbb{X})>H(\mathbb{Y})$ and $H(\mathbb{M})>H(\mathbb{N})$. For some convex ...
1
vote
1answer
58 views

Proving that the entropy is zero given conditional entropies

Let's suppose we have 4 random variables $X,Y,Z$ and $T$ and that the following equations hold about the entropy: $$H(T|X)=H(T)$$ $$H(T|X,Y)=0$$ $$H(T|Y)=H(T)$$ $$H(Y|Z)=0$$ $$H(T|Z)=0$$ Also, the ...
1
vote
0answers
29 views

conditional mutual information

I have a question about mutual information $$I(Z ; T/X,Y) = I(T/X,Y ; Z)$$ $T,X,Y,Z$ are random variables is this statement accurate? if it is true and I know that I(Z;T/X,Y) = H(Z/X,Y) - H(Z/T,X,...
1
vote
1answer
67 views

Entropy and Mutual Information

Consider two discrete random variables $X$ $\{x_1,x_2,\dots,x_n\}$ and $Y$ $\{y_1,y_2,\dots, y_n\}$. Lets say that entropy $H(X)=0$ i.e. $X$ has a probability distribution s.t. $P(X=x_j) = 1$ for only ...
0
votes
1answer
58 views

Conditional entropy under quantization

Let $X$ be a continuous random variable and $X^n$ its quantization that becomes finer with larger $n$. Let $Y$ be a deterministic function of $X$. Then we have that the conditional entropy $$H(Y|X) = ...
0
votes
1answer
30 views

Not understanding steps in derivation for entropy of a Gaussian random variable

Can someone explain the last two steps in the derivation given below? This is the derivation of the entropy of a Gaussian random variable:
0
votes
1answer
35 views

If $X$ to $Y$ to $Z$ is a Markov chain, prove or disprove $H(Y\mid X)\le H(Z\mid X)$

If $X \to Y \to Z$ is a Markov chain, prove or disprove $H(Y\mid X)\le H(Z\mid X)$. I said the statement was true, and from $I(X;Y)\ge I(X;Z)$ by definition, thus $H(X) - H(X\mid Y) \ge H(X)-H(X\mid ...
1
vote
1answer
41 views

Rate distortion function with infinite distortion

I am working through the problems in Elements of Information Theory by Cover and Thomas and have come across the following problem I couldn't answer. The problem is to find the rate distortion ...