This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

0
votes
0answers
15 views

Intuitive explanation of Shannon's source entropy in information / communications theory

I am trying to calculate the number of bits required to encode a message. FOr that, I am applying Shannon's entropy, H. I have done the implementation and playing around with themessage length and ...
1
vote
1answer
23 views

How to test quality of probability estimates?

I have a Markov chain model which produces a probability distribution for absorption in 4 possible absorbing states. I.e. the model estimates the probability distribution for a discrete random ...
1
vote
0answers
9 views

Channel capacity of sum of symmetric channels

I've got a channel matrix $P$ of the form $\begin{bmatrix} Q \\ R \end{bmatrix}$ where $Q,R$ are channel matrices of symmetric channels, so they now have different input alphabets but the ...
0
votes
1answer
21 views

Basic Entropy Inequality and Identity question

This is a solution to a problem I am working on: \begin{equation} \begin{aligned} H(X|Y) + H(Y|Z) &\ge^? H(X|Y, Z) + H(Y|Z) \\ &=^\text{?}H(X,Y |Z) \\ &= H(X|Z) + H(Y|X, Z)\\ &\ge ...
0
votes
0answers
11 views

Entropy of degenerate multivariate normal

After having read the Wikipedia entry on the degenerate case of the multivariate normal distribution: https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case my question is: ...
0
votes
1answer
13 views

Calculating Entropy and Information Gain of a Variable

I have the following values for two random variables. I need to compute the following values: a. H(Y) b. H(Y|X) c. and finally IG(Y|X) I will show what I have calculated so far. a. H(Y) = ...
0
votes
1answer
15 views

Estimation for entropy

Let $T\colon X\to X$ be continuous and $X$ compact and $K\subset X$ compact. By $s_n(2^{-k},K,T)$ denote the maximal cardinality of any $(n,2^{-k})$ separated subset of $K$. Suppose, we know for ...
2
votes
1answer
54 views

proof of upper bound on differential entropy of f(X)

I asked a similar question yesterday, but I organized my question here a little and further asked my second question. Suppose $X$ is a continuous random variable with the pdf $f_x$, and $Y=g(X)$. If ...
0
votes
0answers
30 views

How to calculate the Shannon Entropy for a block length of a word

I have a binary sequence of length N as $10110110111...$ I want to segment the above series into equal blocks of a window of length $L$. One way of determining the block length is using the ...
0
votes
0answers
13 views

pattern-sensible entropy measure

I have some binary images (meaning each pixel can be 0 or 1), I want to find a pattern-sensible entropy measure, which means for example that a chessboard should have a very low entropy value (almost ...
0
votes
0answers
7 views

Can the entropy of a system of multiple stochastic processes be defined?

I'm trying to measure the entropy of BINGO cards. (waiting for laughter to die down.) Normally one would think of this as 75 choose 24, but in reality it is 15 choose 5, four times, and 15 choose 4, ...
0
votes
0answers
3 views

Calculating the entropy of original set E(S)

I am currently revising for an exam and I am struggling with this question Using a decision tree algorithm with information gain splitting, which of the two attributes diarrhea or fever is a ...
0
votes
0answers
11 views

Compare two distributions with varying focus on different regions

I have been trying to find if my problem matches has been discussed in prior research and if any technique exists to solve it. Here's the problem: Given two distributions (pdf) D1 and D2 over a ...
1
vote
1answer
14 views

Entropy of factor map

Let $A$ and $B$ be two compact metric spaces with $B\subset A$- Moreover, let $T\colon A\to A$ continuous and let $S\colon A\to B$ a continuous surjection with $S\circ T=T\circ S$. Moreover assume ...
2
votes
1answer
76 views

Random permutations composition

I'm trying to prove a theorem that seems very intuitive. However, I seem to be missing a piece. If: $\pi$ is a random permutation ($S_n$), $\pi_1, \pi_2$ - random permutations with uniform ...
0
votes
0answers
13 views

Calculating Split Info with given equation not matching solution

I'm given the formula to calculate the Split info but cannot seem to calculate the correct answer of 0.926 that the example shows. Split Info $= -\Sigma \frac{\mid D_j\mid}{\mid D\mid} * log_2 ...
0
votes
1answer
21 views

Decomposition of the entropy

So, I'm reading about this property in the MacKay book. But I don't fully get it. Can someone explain it to me? There's this example: A source produces a character $x$ from the alphabet $A = \{0, ...
3
votes
2answers
59 views

Deriving the q-Gaussian PDF

Ok it may sound a bit too simple but I am quite confused here. While studying generalized entropic forms, in my case that of $S_q$ or in another words the Tsallis Entropy, I reach a point where I have ...
0
votes
1answer
18 views

Calculating mutual information for a dataset

I have a dataset of individual text documents $D = {d_0, d_1, ..., d_n}$ and a corpus of keywords $K = {k_0, k_1, ..., k_m}$ in the documents. There are zero or more keywords in each text document. I ...
1
vote
1answer
37 views

Conditional Entropy and Gibbs Inequality

We know $$H(X | Y) + H(Y) = H(X, Y)$$ Therefore, $$H(X | Y) \leq H(X, Y) $$ since $$ H(Y) \geq 0$$ If we expand this out, we get $$-\sum_{x,y} {p(x,y) \log p(x | y)} \leq - \sum_{x,y} {p(x,y) \log ...
0
votes
1answer
31 views

Entropy/Variance inequality

The following inequality is sometimes used as a building block to prove log Sobolev inequalities. Does anyone have a simple proof of it? $$ x\log x + y\log y - (x+y)\log \frac{x+y}{2}\leq (\sqrt ...
1
vote
1answer
26 views

Convert a joint entropy matrix to a contidional entropy matrix.

I've only barely started to learn about Entropy and Information Theory as a part of a course I'm taking in Systems Theory / Cybernetics. The thing is, I'm terrible at math! Say I have a joint ...
2
votes
1answer
50 views

Password entropy of famous xkcd comic

The famous xkcd comic about password strength calculates the entropy of the 11-character password "Tr0ub4dor&3" with 28 bits of entropy. When following the ASCII-95-chart, we have 95 possible ...
1
vote
1answer
29 views

Does entropy inequality hold for convex combination

I have two pairs of Random Variables, $(\mathbb{X},\mathbb{Y})$ and $(\mathbb{M},\mathbb{N})$ which satisfies, $H(\mathbb{X})>H(\mathbb{Y})$ and $H(\mathbb{M})>H(\mathbb{N})$. For some convex ...
1
vote
1answer
55 views

Proving that the entropy is zero given conditional entropies

Let's suppose we have 4 random variables $X,Y,Z$ and $T$ and that the following equations hold about the entropy: $$H(T|X)=H(T)$$ $$H(T|X,Y)=0$$ $$H(T|Y)=H(T)$$ $$H(Y|Z)=0$$ $$H(T|Z)=0$$ Also, the ...
1
vote
0answers
24 views

conditional mutual information

I have a question about mutual information $$I(Z ; T/X,Y) = I(T/X,Y ; Z)$$ $T,X,Y,Z$ are random variables is this statement accurate? if it is true and I know that I(Z;T/X,Y) = H(Z/X,Y) - ...
1
vote
1answer
61 views

Entropy and Mutual Information

Consider two discrete random variables $X$ $\{x_1,x_2,\dots,x_n\}$ and $Y$ $\{y_1,y_2,\dots, y_n\}$. Lets say that entropy $H(X)=0$ i.e. $X$ has a probability distribution s.t. $P(X=x_j) = 1$ for only ...
0
votes
1answer
52 views

Conditional entropy under quantization

Let $X$ be a continuous random variable and $X^n$ its quantization that becomes finer with larger $n$. Let $Y$ be a deterministic function of $X$. Then we have that the conditional entropy $$H(Y|X) = ...
0
votes
1answer
29 views

Not understanding steps in derivation for entropy of a Gaussian random variable

Can someone explain the last two steps in the derivation given below? This is the derivation of the entropy of a Gaussian random variable:
0
votes
1answer
34 views

If $X$ to $Y$ to $Z$ is a Markov chain, prove or disprove $H(Y\mid X)\le H(Z\mid X)$

If $X \to Y \to Z$ is a Markov chain, prove or disprove $H(Y\mid X)\le H(Z\mid X)$. I said the statement was true, and from $I(X;Y)\ge I(X;Z)$ by definition, thus $H(X) - H(X\mid Y) \ge H(X)-H(X\mid ...
0
votes
0answers
22 views

Rate distortion function with infinite distortion

I am working through the problems in Elements of Information Theory by Cover and Thomas and have come across the following problem I couldn't answer. The problem is to find the rate distortion ...
1
vote
0answers
26 views

Entropy of a dictionary

I have an english dictionary (a file that contains a list of words) and I want to calculate: given a path tree (a word), measure $H(C_{l+1}|C_l=c_l, C_{l-1}=c_{l-1}, ...)$ for a few levels of the ...
0
votes
0answers
21 views

Different methodology for maximizing entropy in continuous random variable case

Suppose we want to maximize the well-known Shannon entropy $S=-∫_{0}^{x_{max}}f(x)lnf(x)dx$ subject to the following constraints $∫_{0}^{x_{max}}f(x)dx=1$, $∫_{0}^{x_{max}}xf(x)dx=x ̅$ and so on ...
-1
votes
1answer
14 views

I need assistance with Information Gain

This might not be mutual to mathematics but it does relate to Information-Theory. My question is that I don't understand whether the InformationGain algorithm, in Decision-Tree machine-learning, ...
0
votes
0answers
29 views

Channels with memory have higher capacity

I am working through Elements of Information Theory by Cover and Thomas and have come across the following solution to one of their problems that I don't understand. Consider a binary, symmetric ...
1
vote
0answers
14 views

Entropy of first char of a word in dictionary

Suppose I have an English dictionary, that is a list of words in a file. I have to calculate the entropy of the first char. I calculated the probability of each first char ($P_a, P_b, \dots, P_z$) in ...
5
votes
1answer
181 views

Is there a possibility to determine/ estimate the topological entropy?

By $E$, denote the set of excited states $E=\left\{1,2,\ldots,e\right\}$ and by $R$ the set of refractory states $R=\left\{e+1,e+2,\ldots,e+r\right\}$. By $0$, denote the equilibrium state. The ...
1
vote
1answer
25 views

Entropy Inequality $H(X| g(Y))\geq H(X|Y)$

Let $X,Y$ be random discrete variables. $H(X) = -\sum\limits_{x}P\{X=x\}\operatorname{log}_2P\{X=x\}$ be the entropy-function. It is known fact that that $H(g(Y))\leq H(Y)$. I want to prove the ...
6
votes
0answers
55 views

At what rate does the entropy of shuffled cards converge?

Consider a somewhat primitive method of shuffling a stack of $n$ cards: In every step, take the top card and insert it at a uniformly randomly selected one of the $n$ possible positions above, between ...
0
votes
1answer
33 views

Existence of measure(s) of maximal entropy, given a finite-to-one chaotic global attractor $A$ which is, moreover, the non-wandering set

Let $X$ be a compact metric space and let $f\colon X\to X$ define some dynamics, $f$ being continuous and finite-to-one on a global attractor $A\subset X$. Moreover, $A$ is the non-wandering set ...
0
votes
0answers
37 views

Is my entropy calculation correct? Clustering entropy example

I would like to calculate entropy of this example scheme http://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-clustering-1.html Equation of entropy Then the entropy is (the first line) ...
4
votes
1answer
90 views

Mutual information vs Information Gain

I always thought that mutual information and information gain refer to the same thing, however looking at Wikipedia: http://en.wikipedia.org/wiki/Information_gain ...
0
votes
0answers
11 views

Is there an obvious explanation for why the entropy formula is better than standard deviation as a numerical summary for spread of a distribution?

I'm currently reading Murphy's Machine Learning book, and he briefly introduces the notion of entropy for a random variable. He states that the entropy is $$ \mathbb{H}(X)=-\int p(x)\log_{2}p(x)\,dx, ...
0
votes
1answer
29 views

Entropy of gamma-exponential compound distribution

Following this question, I have the PDF of a gamma-exponential compound distribution as $$f(y) = \frac{\alpha\beta^{\alpha}} {(y+\beta)^{(\alpha+1)}} $$ For my application I need the entropy of ...
0
votes
0answers
55 views

Generalization of Shannon's source coding theorem with a posteriori entropies

This doubt is with reference to section 5-5 of "Information theory and Coding" by Prof. Norman Abramson. Under the topic "A generalization of Shannon's First Theorem", the text discusses how knowledge ...
2
votes
0answers
20 views

Showing that for a family of subsets of $[n]$ enough elements appear in high frequencies

Let $\mathcal{F} \subseteq 2^{[n]}$ a familiy of subsets. Assume that the following applies: For every $A \subseteq [n]$ , such that $|A|\leq \alpha n$ ($\alpha > 0$ is given), there's a subset ...
1
vote
1answer
50 views

Calculating the capacity of a Binary Channel?

I'm pretty new in information theory so I need your suggestions or references in order to solve this problem. I have my attempt below. Problem: Every second, Alice can either send (bit 1) or not ...
0
votes
1answer
24 views

PRNG for compression

I'm trying to intuitively grasp information theory. You have a string of size X that contains a lot of information, say it's a movie. You have a string of size N << X which is going to be the ...
0
votes
1answer
48 views

Conditional entropy and independent conditioning variables

Let $X,Y,Z,Y',Z'$ be random variables where $Y\sim Y', Z\sim Z'$, $Y$ and $Z$ are independent, while $Y'$ and $Z'$ are, in the sense that we have $p(X,Y,Z)=p(X|Y,Z)p(Y)p(Z)$ ...
1
vote
0answers
38 views

mutual information and data processing inequality for $X\to Y\to Z$ where $Y=f(X)$

Let $X\to Y\to Z$ be three random variables. The data processing inequality states $I(X;Y)\geq I(X;Z)$. Further assume $Y=f(X)$ where $f:\mathcal{X}\to\mathcal{Y}$ is an arbitrary function. What ...