Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [entropy]

This tag is for questions about mathematical entropy. If you have a question about thermodynamical entropy, visit Physics Stack Exchange or Chemistry Stack Exchange instead.

1
vote
2answers
16 views

“triangular” relation conditional entropy

Let $X,Y,Z$ be random variables. Assume we know the conditional entropies $H(Y|X)$ and $H(Z|Y)$ but we want to bound $H(Z|X)$ which is unknown Is there any relation between these 3 quantities, some "...
2
votes
1answer
29 views

Entropy Theory and Conditional Entropy

Peter Walters An Introduction to Ergodic Theory. Chapter 4 Entropy Page 83 & 84 How did they duduce the last formula ?
4
votes
1answer
98 views

Maximum Entropy Distribution with Reciprocal Symmetry

What is the maximum entropy distribution $F$ on $(0,\infty)$ with mean $1$ and $\Pr(x\le a)= \Pr(x\ge\frac{1}{a})$ for all $a$? After taking a derivative we find that the pdf $F'=f$ must satisfy $af(...
0
votes
0answers
19 views

Total variation distance and entropy

Let $X$, $Y$ be discrete random variables taking values within the same set of $N$ elements. Let the total variation distance $|P-Q|$ (which is half the $l_1$ distance between the distributions of $P$ ...
24
votes
4answers
2k views

Shannon entropy of a fair dice

The formula for Shannon entropy is as follows, $$\text{Entropy}(S) = - \sum_i p_i \log_2 p_i $$ Thus, a fair six sided dice should have the entropy, $$- \sum_{i=1}^6 \dfrac{1}{6} \log_2 \dfrac{1}{6}...
0
votes
0answers
34 views

Entropy of sorted card deck

My teacher says entropy is measured in bits of information (you get 1 bit of information if you know the result of a coin flip). Now the other question is how many bits of information you get from ...
3
votes
2answers
37 views

inequality between entropy measure and Herfindahl index

I am considering the entropy measure and the Herfindahl-Hirschmann- (or Hirschmann-Herfindahl)-Index to measure diversification of firms. A firm is perfectly focused if it generates all sales in one ...
4
votes
0answers
66 views

Properties of the differential entropy of Hermite functions

Preliminaries: let $f_n(x)\,, n=0,\ldots,\infty$ denote the Hermite functions, which are of the form $$ f_n (x)=\#\, e^{-\frac{x^2}{2}} H_n(x) \,,$$ where $H_n(x)$ are the physicists' Hermite ...
2
votes
2answers
29 views

How to construct an example for the entropy equation: $H(Z)=H(X)+H(Y)$ where $Z=X+Y$ [duplicate]

Given $Z=X+Y$ where X and Y are two random variables, under what conditions does $H(Z)=H(X)+H(Y)$? Notice $Z$ is a function of $(X,Y)$, therefore $H(Z)\leq H(X,Y)$, and $H(X,Y)\leq H(X)+H(Y)-I(X;Y)$. ...
0
votes
0answers
16 views

What's the entropy of a time-series?

Entropy is defined as $H(X)=\sum\limits_{x \in X} p(x)log p(x)$ and is usually used for measuring the uncertainty of a system. I wonder if the entropy concept can be applied to time series ? If so ...
0
votes
2answers
33 views

Why the solution to this entropy problem is not simply $H(X)=-a*log(a) - (1-a)*log(1-a)$?

Let $X_1$ and $X_2$ be discrete random variables drawn according to probability mass function $p_1$ and $p_2$ over the respective alhabets $X_1={1,2,...m}$ and $X_2={m+1,...,n}$. Let $X=X_1$ with ...
1
vote
1answer
34 views

Alternative ways of sampling from a distribution

I have recently been working on some numerical algorithm that required me to pick a random element $r_i$ from a finite set $R$ with probability $p_i$. This is a fairly standard procedure and many ...
0
votes
0answers
38 views

What is the simplest way to show that the following inequality is true in information theory?

I have following relation of random variables $$Y_1=aX_1+bX_2+N_1,\\Y_2=X_1+X_2+N_2,$$ where $X_1,X_2$ are discrete random variables which can take a value uniformly from a set and $N_1,N_2$ are ...
1
vote
1answer
39 views

How to show the following relation is true in information theory?

Suppose $X_1,X_2$ are two random variables which can take values from the set $\mathcal{X}$ with uniform distribution. Further, $N$ is a Guassian random variable with zero mean and unit variance. In ...
0
votes
1answer
21 views

How to proof the Fano's inequality using the following formulation?

The Fano's inequality for the Markov chain $X\to Y\to \hat{X}$ is given as follows $$H(X|\hat{X})\leq H(E)+P(E)\log_2(|\chi|),$$ where $E$ is error random variable defined such that $E=1$ if $X\neq \...
0
votes
1answer
28 views

What is the simplified formula to calculate joint conditional entropy of 4 or more variables for instance H(a|b,c,d) or H(a|b,c,d,e)?

I'm a medical science student and I came across a point in my research I've to work with entropy. As entropy involve probability theory which , like many, I'm not good at ;) I can calculate an H(a) ...
3
votes
1answer
69 views

Pattern Recognition and Machine Learning (Bishop) - Exercise 1.28

1.28 In Section 1.6, we introduced the idea of entropy $h(x)$ as the information gained on observing the value of a random variable $x$ having distribution $p(x)$. We saw that, for independent ...
1
vote
0answers
38 views

How to analytically compute KL divergence of two Gaussian distributions?

Consider two multi-variate Gaussian distributions, $p(x)=\mathcal N(x;\mu_p, \sigma_p^2)$ and $q(x)=\mathcal N(x; \mu_q, \sigma_q^2)$. It seems the KL-divergence of these two Gaussian distributions $...
1
vote
1answer
40 views

Find probability distribution given constraints?

I am looking at the following problem. I have a function $f(x)$ with support on $[0, \infty)$. Furthermore, $f(x)$ is bounded between 0 and 1, monotonically increasing and concave everywhere. ...
3
votes
2answers
85 views

Differential entropy vs Kolmogorov-Sinai “partition trick”

Shannon entropy is well-defined for probability distributions $p(x)$ on finite (or countable) sets $X$, \begin{equation} H_S=-\sum_{x\in X}p(x)\log p(x)\,. \end{equation} To compute the entropy of a ...
2
votes
1answer
49 views

How do I find the set of functions that would make this non-linear operator diverge?

I have this non linear operator $$H(p) = -\sum_{n=0}^ {\infty} p_n ln(p_n)$$ where $p_n$ are given by a function $p(n)$ when $n$ is a whole number. I want to find what set of $p(n)$ makes $H(p)$ ...
2
votes
0answers
50 views

Lower bound for quantum relative entropy

In my research this summer, I have become interested in lower bounds on the standard "Umegaki quantum relative entropy". For two non-negative matrices $X$ and $Y$, the Umegaki quantum relative ...
0
votes
1answer
30 views

Comparing Differential Entropy of Normal Distribution to Shannon Entropy

The (differential) entropy of the multivariate normal distribution is given by: $$H(\underline{X}) = \frac12 \ln(|2 \pi e \Sigma|)$$ Does the Shannon entropy: $$ H(\underline{X})=−p(\underline{x}...
3
votes
0answers
58 views

Improving on the data processing inequality for Markov chains

Given a Markov chain $X \rightarrow Y \rightarrow Z$, the data processing inequality bounds the mutual information $$I(X;Z) \leq \min \big( I(X;Y),I(Y;Z) \big)$$ However, it seems intuitive that we ...
0
votes
0answers
27 views

What is the Shannon entropy of a quantized normal distribution?

The familiar Shannon entropy of an independent and identical distribution is:- $$ -\sum_i p_i\log_2(p_i) $$ I have a discrete distribution of something like:- I can model it with a quantized normal ...
0
votes
1answer
42 views

Lossless Compression and definition of Entropy

I am taking an intro class to Information Theory and I have a question. Suppose we have three symbols a, b, and c with probabilities of them coming out of a chanell ...
1
vote
0answers
27 views

Entropy of dyadic toeplitz system

I have failed to find the topological entropy of dyadic Toeplitz system. Do you know what this entropy is? Dyadic Toeplitz system is a subshift of $\{0,1\}^{\mathbb{Z}}$, i.e. it is an orbit closure ...
0
votes
0answers
23 views

need help identifying a formula for “pseudo-entropy”

Maintaining some old code, I've come across: $$\text{pseudo-entropy} = -x \log(x) + x ^{0.45} \cdot (1 - x) ^ {16}$$ I simply need a name for this formula so I can read up on what it's supposed to ...
1
vote
1answer
38 views

Proof of $H(Y|X)=H(Y)$ when X and Y are independent

Does anyone know how to prove the $H(Y|X)=H(Y)$ when X and Y are independent? I know the proof of $H(Y|X)=\sum p(x,y)\log_2p(y|x)$,but I found that I can't prove $H(Y|X)=H(Y)$ when $X$ and $Y$ are ...
0
votes
0answers
54 views

How to find log of “sum of two matrices”?

I want to find log (A + B ) where A and B are matrices. The context is that I want to find the Von Neumann entropy which is given by: $Entropy = - Trace [\rho log (\rho) ]$ where $\rho$ is a matrix....
2
votes
1answer
66 views

Under what conditions does $H(X\mid f(Y))=H(X\mid Y)$?

I have the problem that I cannot solve: Under what conditions does $H(X∣f(Y))=H(X∣Y)$? I would like to draw a result about the relation between $p_X(\cdot | g(Y))$ and $p_X(\cdot | Y)$. Are they equal?...
1
vote
0answers
33 views

Entropy conditioned on a function of a r.v

I have problem in proving $H(Y \mid X) \leq H(Y \mid f(X))$, where $f$ is a function of $X$. In the textbooks, they already proved $I(Y; X) \geq I(Y;f(X))$, and $H(f(X)) \leq H(X)$ but I can't relate ...
0
votes
1answer
41 views

conditional probability which their condition have $XOR and Z=X+Y$

Let $X$ and $Y$ be two independent binary random variable with the same alphabet {$0,1$},ie,$Pr(0)=Pr(1)=\frac{1}{2}$ Define $I(X;Y|Z)=H(X|Z)-H(X|Y,Z)$ $1.$Let $Z=X+Y$,Find $I(X;Y|Z)$ $2.$Let $Z=X ⊕...
0
votes
1answer
38 views

How to evaluate the change of information of a random variable?

Given a random variable $X$ having finite alphabet $\mathcal{A}_X$ and valid $p_x(\cdot)$ (for which there is no $x_0 \in \mathcal{A}_X$ so that $p_x(x_0) = 0$) I want to know its actual outcome (...
0
votes
2answers
52 views

Priors in Shannon and Rényi entropies

[Note: Cross-posted at Cross Validated StackExchange] I am new to information theory and currently working with Shannon and Rényi entropies. Given the pdf $p_{\theta}(x)$ of a random variable $x$, ...
2
votes
1answer
101 views

entropy of the sum of binomial distributions

Suppose that one has $X_1 \sim Bin(n,p)$ and $X_2 \sim Bin(n,1-p)$ and that $Z$ is distributed s.t: $$ P(Z = k) = .5 P(X_1=k) + .5P(X_2=k) $$ How do we compute the entropy? For a binomial ...
1
vote
1answer
54 views

Known Markov-type inequalities on entropy?

Let $P$ be discrete distribution where the $i$-th value has probability $p_i$. Define $H$ as the random variable returning $-\log(p_i)$ with probability $p_i$. By definition, ${E}(H) = \mathrm{H}(P)$....
0
votes
0answers
41 views

data processing inequality-mutual information

suppose that we have a family of probability mass functions ${f_\theta }\left( x \right)$ indexed by $\theta$, and let $x$ be a sample from this distribution. Then from the information theory, we have ...
4
votes
2answers
72 views

Entropy of a Measure Preserving Transformation

$\newcommand{\set}[1]{\{#1\}}$ $\newcommand{\lrp}[1]{\left(#1\right)}$ I am reading the concept of entropy from Peter Walters An Introduction to Ergodic Theory and I am having trouble understanding ...
0
votes
1answer
44 views

Measure of information in a particular example

Consider one experiment with two possible results described by a random variable $X$ attaining values $x_1 = 0$ or $x_2 = 1$ to capture the two possible outcomes. Every such random variable is ...
0
votes
3answers
62 views

How can they come up with the definition of entropy in information theory? [duplicate]

I have read some books about information theory but I don't have any ideas how can they find the definition of entropy? We have $$H(X)=-\sum_{x\in X}p(x)\, \text{log}\, p (x)$$ X is a discrete random ...
1
vote
1answer
49 views

From combinatorial entropy to Shannon entropy

This blog, Shannon entropy, by Yurii Lahodiuk shows the link (derivation) of Shannon entropy from basic combinatorics. I would like to know the first person that made this combinatorial interpretation ...
0
votes
0answers
54 views

Parameter meanings in Rényi relative entropy and Tsallis relative entrop

There are three types of relative entropy that are used in financial studies, Kullback_Leibler relative entropy: $H_k(Q,P)=E^p\left(\frac{dQ}{dP} \ln\frac{dQ}{dP}\right)$ Rényi relative entroy: $H^...
1
vote
1answer
21 views

Equality of entropy $\iff$ Same probabilities under permutation?

Assume I have: $$H(p_1,\ldots,p_N)=H(q_1,\ldots,q_N)$$ where $H$ is the Shannon Entropy. Does that mean that I necessarily have the $p_i$ and $q_i$ linked by a permutation? Or is it not true? For ...
0
votes
1answer
26 views

Relationship between mutual information and entropy

I know that for two system $X$ and $Y$, we can write : $$ H(X,Y)=H(X)+H(Y)-I(X,Y)$$ Where $I$ is called the mutual information and $H$ is the shannon entropy. My question is : do we have another ...
1
vote
1answer
75 views

Entropy of a countable nested intersection of SFTs

Let ${\scr F} = \{w_1, w_2,\dots\}$ be a countable collection of words in some finite alphabet, and let $X$ denote the subshift of bi-infinite strings that avoid the elements of $\scr{F}$. For each ...
0
votes
0answers
30 views

Entropy lower bound

I have the following observation: $$Z=\sqrt{P}H+W,$$ where $$H \sim \mathcal{N} (0,\sigma^2). $$ $P$ is given and fixed. I am trying to find $W$ that minimizes $h(Z)$. Given that $E[|W|^2]=\sqrt{\...
0
votes
0answers
28 views

Bound $E|x_1+\cdots+x_n-n/2|$ for $n$ odd and $x_i$ iid uniform bits using Pinsker inequality?c

This is an exercise (not a homework) from https://homes.cs.washington.edu/~anuprao/pubs/book.pdf. The title is a rephrasing I did. The original is let $n$ be odd, we take a random $n$ bit string out ...
0
votes
0answers
26 views

Calculating entropy of known sequence like 0,1,2,3 …

If we apply Shannon's formula to a sequence of numbers that we know how to generate (for example, natural numbers), shouldn't entropy be 0? Mine might be a too intuitive definition of entropy. But if ...
0
votes
2answers
16 views

How to calulate entropy $H(X+S+Z,X+\alpha S)$

I have no idea how to calulate entropy $H(X+S+Z,X+\alpha S)$ where $X\sim N(0,P)$, $S\sim N(0,Q)$, $Z\sim N(0,N)$ and $\alpha$ is a constant number.Here we use $N$ to denote Gaussian distribution. Can ...