Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [entropy]

This tag is for questions about mathematical entropy. If you have a question about thermodynamical entropy, visit Physics Stack Exchange or Chemistry Stack Exchange instead.

0
votes
0answers
6 views

Same max entropy for diffirent priors

For a continuous distribution,$f$, we define the entropy with respect to a reference prior $f_{0}$ to be $$\epsilon(f)=\int \log(\frac{f(\theta)}{f_{0}(\theta)})f_{0} d\theta$$ For Lebesgue measure ...
0
votes
0answers
24 views

Lebesgue Measure and absolute continuity

I read that a measure $m_{1}$ is said to be absolutely continuous with respect to another measure, $m_{2}$ if $m_{1}(S)=0$ whenever $m_{2}(S)=0$ for any set $S$. So if $m_{1}$ was Lebesgue measure, ...
4
votes
0answers
48 views

Can this vague feeling about entropy be made precise?

Let $I$ denote the unit interval and $\mu$ be the Lebesgue measure. Let $S:I\to I$ be the map defined as $S(x)=2x \pmod{1}$. Then it is known that for any measurable subset $A$ of $I$ we have $$ \lim_{...
1
vote
1answer
35 views

True random generation given radioactive entropy

I have been reading various sources for experimentation with truly random numbers. As I understand, it is impossible for a computer to generate a "truly random" number as they are deterministic in ...
2
votes
1answer
24 views

Partial derivative of cross-entropy

I am trying to make sense of this question. $$E(t,o)=-\sum_j t_j \log o_j$$ How did he derive the following? $$\frac{\partial E} {\partial o_j} = \frac{-t_j}{o_j}$$
0
votes
0answers
16 views

Nice dimension independent proof of entropy inequality?

How does one prove that for arbitrary finite number of random variables $X,Y,Z,T,...$ the Shannon Entropy inequality holds $H(X,Y,Z,T,...) \leq H(X)+H(Y)+H(Z)+H(T)+...$ I know how to do it with ...
0
votes
1answer
15 views

Why doesn't the law of total expectation apply here? $E[X] = E[E[X \mid Y]]$

I'm learning about Entropy for the first time. From Wikipedia, $$H(Y \mid X = x) = E[I(Y) \mid X=x]$$ and the confusing part for me is this statement: " $H(Y \mid X)$ is the result of averaging $H(Y ...
0
votes
0answers
23 views

Learning about Entropy, notation question: is $P(X)$ a random variable?

Background I'm taking a first year probability course and I've gotten reasonably comfortable calculating $E[g(X)]$ using LOTUS. Now I'm learning about Entropy and for the first time I'm encountering ...
0
votes
0answers
26 views

Mean and variance of empirical entropy

$M$ samples are drawn from a discrete probability distribution $\bigl( p_i>0; \; \sum_{i=1}^{N} p_i \bigr)$. The number of times result number $i$ has occurred is given by $M_i$ (s.t. $\sum_{i=1}^N ...
0
votes
0answers
12 views

How can I solve this entropy example

X is number of points for throwing a fair dice X can be 1,2,3,4,5,6. and Y = 1 if X <5 , Y = 0 if X = 6. Then, what is the answer of H(Y) and H(X|Y)
0
votes
0answers
19 views

The integration of the entropy to the exponential distribution

I would like to ask the integration of the entropy to the exponential distribution, which is as below: Exponential: $$ p_{x}(x)=\theta*e^{-{\theta}*x}$$ ...
0
votes
1answer
17 views

How much BPS(Bits per symbol) is enough to call a compression algorithm good, with respect to entropy?

Consider a general purpose lossless data compression algorithm, It compresses a randomly generated binary file of 100MB size, with random I mean I wrote a small Script to create a file with random ...
0
votes
0answers
30 views

Entropy of random variable

Let $(\Omega, \mathcal{F}, P)$ be a probability space with $|\Omega|=m <\infty$ and $P(\omega)>0 \ \forall \omega \in \Omega$. $H(P):=-\int \log(P(\omega))dP(\omega)$ denotes the differential ...
0
votes
0answers
38 views

Prove $H(X,Y) ≤ H(X) + H(Y)$

I am trying to prove that $H(X,Y) \leq H(X) + H(Y)$ where $X,Y$ are two random variables. I saw that someone else posted this question before as well, but the discussion about it wasn't quite what I ...
0
votes
0answers
33 views

Entropy maximizing distribution when power constraint exists for one random variable

Suppose I have a random vector $\bar{X}=[X_{1},X_{2}, X_{3}]$. X1,X2 and X3 can take values from the alphabet {0,1,2,3} . (This can be even generalized to a finite set of cardinality N). I don't have ...
0
votes
1answer
42 views

Find probability distribution (joint pdf) that satisfy constraints

Suppose I have a random vector $\bar{X}=[X_{1},X_{2}]$. $X_{1}$ and $X_{2}$ comes from the alphabet {0,1,2}. I don't have any information on the probability distribution of these random variables. In ...
0
votes
3answers
93 views

What is the relationship between information in the sense of Shannon entropy and information for the human brain?

In an informatic theoretic sense, complete randomness maximizes information. For instance, an image of randomly distributed black and white pixels has a very high entropy/information. For a human ...
0
votes
0answers
30 views

Kolmogorov-Sinai theorem for a generator with infinite entropy

I can find many books and documents stating the Kolmogorov-Sinai theorem (that is $h(T) = h(T,P)$ if $P$ is a generating partition) when the generating partition is finite or countable with finite ...
2
votes
2answers
56 views

Does the Shannon Entropy always exist (even for infinite distributions)?

Let $p : \mathbb{N} \to [0, 1]$ be a probability distribution over the naturals. The Shannon Entropy is: $$H = -\sum_{n=0}^\infty p(n)\log_2 p(n)$$ Does this series always converge? I tried a ...
2
votes
2answers
34 views

“triangular” relation conditional entropy

Let $X,Y,Z$ be random variables. Assume we know the conditional entropies $H(Y|X)$ and $H(Z|Y)$ but we want to bound $H(Z|X)$ which is unknown Is there any relation between these 3 quantities, some "...
2
votes
1answer
38 views

Entropy Theory and Conditional Entropy

Peter Walters An Introduction to Ergodic Theory. Chapter 4 Entropy Page 83 & 84 How did they duduce the last formula ?
4
votes
1answer
103 views

Maximum Entropy Distribution with Reciprocal Symmetry

What is the maximum entropy distribution $F$ on $(0,\infty)$ with mean $1$ and $\Pr(x\le a)= \Pr(x\ge\frac{1}{a})$ for all $a$? After taking a derivative we find that the pdf $F'=f$ must satisfy $af(...
0
votes
0answers
28 views

Total variation distance and entropy

Let $X$, $Y$ be discrete random variables taking values within the same set of $N$ elements. Let the total variation distance $|P-Q|$ (which is half the $l_1$ distance between the distributions of $P$ ...
25
votes
4answers
2k views

Shannon entropy of a fair dice

The formula for Shannon entropy is as follows, $$\text{Entropy}(S) = - \sum_i p_i \log_2 p_i $$ Thus, a fair six sided dice should have the entropy, $$- \sum_{i=1}^6 \dfrac{1}{6} \log_2 \dfrac{1}{6}...
0
votes
0answers
35 views

Entropy of sorted card deck

My teacher says entropy is measured in bits of information (you get 1 bit of information if you know the result of a coin flip). Now the other question is how many bits of information you get from ...
3
votes
2answers
43 views

inequality between entropy measure and Herfindahl index

I am considering the entropy measure and the Herfindahl-Hirschmann- (or Hirschmann-Herfindahl)-Index to measure diversification of firms. A firm is perfectly focused if it generates all sales in one ...
4
votes
0answers
72 views

Properties of the differential entropy of Hermite functions

Preliminaries: let $f_n(x)\,, n=0,\ldots,\infty$ denote the Hermite functions, which are of the form $$ f_n (x)=\#\, e^{-\frac{x^2}{2}} H_n(x) \,,$$ where $H_n(x)$ are the physicists' Hermite ...
2
votes
2answers
50 views

How to construct an example for the entropy equation: $H(Z)=H(X)+H(Y)$ where $Z=X+Y$ [duplicate]

Given $Z=X+Y$ where X and Y are two random variables, under what conditions does $H(Z)=H(X)+H(Y)$? Notice $Z$ is a function of $(X,Y)$, therefore $H(Z)\leq H(X,Y)$, and $H(X,Y)\leq H(X)+H(Y)-I(X;Y)$. ...
0
votes
0answers
16 views

What's the entropy of a time-series?

Entropy is defined as $H(X)=\sum\limits_{x \in X} p(x)log p(x)$ and is usually used for measuring the uncertainty of a system. I wonder if the entropy concept can be applied to time series ? If so ...
0
votes
2answers
33 views

Why the solution to this entropy problem is not simply $H(X)=-a*log(a) - (1-a)*log(1-a)$?

Let $X_1$ and $X_2$ be discrete random variables drawn according to probability mass function $p_1$ and $p_2$ over the respective alhabets $X_1={1,2,...m}$ and $X_2={m+1,...,n}$. Let $X=X_1$ with ...
1
vote
1answer
36 views

Alternative ways of sampling from a distribution

I have recently been working on some numerical algorithm that required me to pick a random element $r_i$ from a finite set $R$ with probability $p_i$. This is a fairly standard procedure and many ...
0
votes
0answers
41 views

What is the simplest way to show that the following inequality is true in information theory?

I have following relation of random variables $$Y_1=aX_1+bX_2+N_1,\\Y_2=X_1+X_2+N_2,$$ where $X_1,X_2$ are discrete random variables which can take a value uniformly from a set and $N_1,N_2$ are ...
1
vote
1answer
44 views

How to show the following relation is true in information theory?

Suppose $X_1,X_2$ are two random variables which can take values from the set $\mathcal{X}$ with uniform distribution. Further, $N$ is a Guassian random variable with zero mean and unit variance. In ...
0
votes
1answer
30 views

How to proof the Fano's inequality using the following formulation?

The Fano's inequality for the Markov chain $X\to Y\to \hat{X}$ is given as follows $$H(X|\hat{X})\leq H(E)+P(E)\log_2(|\chi|),$$ where $E$ is error random variable defined such that $E=1$ if $X\neq \...
0
votes
1answer
30 views

What is the simplified formula to calculate joint conditional entropy of 4 or more variables for instance H(a|b,c,d) or H(a|b,c,d,e)?

I'm a medical science student and I came across a point in my research I've to work with entropy. As entropy involve probability theory which , like many, I'm not good at ;) I can calculate an H(a) ...
3
votes
1answer
96 views

Pattern Recognition and Machine Learning (Bishop) - Exercise 1.28

1.28 In Section 1.6, we introduced the idea of entropy $h(x)$ as the information gained on observing the value of a random variable $x$ having distribution $p(x)$. We saw that, for independent ...
1
vote
0answers
42 views

How to analytically compute KL divergence of two Gaussian distributions?

Consider two multi-variate Gaussian distributions, $p(x)=\mathcal N(x;\mu_p, \sigma_p^2)$ and $q(x)=\mathcal N(x; \mu_q, \sigma_q^2)$. It seems the KL-divergence of these two Gaussian distributions $...
1
vote
1answer
42 views

Find probability distribution given constraints?

I am looking at the following problem. I have a function $f(x)$ with support on $[0, \infty)$. Furthermore, $f(x)$ is bounded between 0 and 1, monotonically increasing and concave everywhere. ...
3
votes
2answers
95 views

Differential entropy vs Kolmogorov-Sinai “partition trick”

Shannon entropy is well-defined for probability distributions $p(x)$ on finite (or countable) sets $X$, \begin{equation} H_S=-\sum_{x\in X}p(x)\log p(x)\,. \end{equation} To compute the entropy of a ...
2
votes
1answer
50 views

How do I find the set of functions that would make this non-linear operator diverge?

I have this non linear operator $$H(p) = -\sum_{n=0}^ {\infty} p_n ln(p_n)$$ where $p_n$ are given by a function $p(n)$ when $n$ is a whole number. I want to find what set of $p(n)$ makes $H(p)$ ...
2
votes
0answers
59 views

Lower bound for quantum relative entropy

In my research this summer, I have become interested in lower bounds on the standard "Umegaki quantum relative entropy". For two non-negative matrices $X$ and $Y$, the Umegaki quantum relative ...
0
votes
1answer
37 views

Comparing Differential Entropy of Normal Distribution to Shannon Entropy

The (differential) entropy of the multivariate normal distribution is given by: $$H(\underline{X}) = \frac12 \ln(|2 \pi e \Sigma|)$$ Does the Shannon entropy: $$ H(\underline{X})=−p(\underline{x}...
3
votes
0answers
72 views

Improving on the data processing inequality for Markov chains

Given a Markov chain $X \rightarrow Y \rightarrow Z$, the data processing inequality bounds the mutual information $$I(X;Z) \leq \min \big( I(X;Y),I(Y;Z) \big)$$ However, it seems intuitive that we ...
0
votes
0answers
27 views

What is the Shannon entropy of a quantized normal distribution?

The familiar Shannon entropy of an independent and identical distribution is:- $$ -\sum_i p_i\log_2(p_i) $$ I have a discrete distribution of something like:- I can model it with a quantized normal ...
0
votes
1answer
43 views

Lossless Compression and definition of Entropy

I am taking an intro class to Information Theory and I have a question. Suppose we have three symbols a, b, and c with probabilities of them coming out of a chanell ...
1
vote
0answers
27 views

Entropy of dyadic toeplitz system

I have failed to find the topological entropy of dyadic Toeplitz system. Do you know what this entropy is? Dyadic Toeplitz system is a subshift of $\{0,1\}^{\mathbb{Z}}$, i.e. it is an orbit closure ...
0
votes
0answers
23 views

need help identifying a formula for “pseudo-entropy”

Maintaining some old code, I've come across: $$\text{pseudo-entropy} = -x \log(x) + x ^{0.45} \cdot (1 - x) ^ {16}$$ I simply need a name for this formula so I can read up on what it's supposed to ...
1
vote
1answer
39 views

Proof of $H(Y|X)=H(Y)$ when X and Y are independent

Does anyone know how to prove the $H(Y|X)=H(Y)$ when X and Y are independent? I know the proof of $H(Y|X)=\sum p(x,y)\log_2p(y|x)$,but I found that I can't prove $H(Y|X)=H(Y)$ when $X$ and $Y$ are ...
0
votes
0answers
58 views

How to find log of “sum of two matrices”?

I want to find log (A + B ) where A and B are matrices. The context is that I want to find the Von Neumann entropy which is given by: $Entropy = - Trace [\rho log (\rho) ]$ where $\rho$ is a matrix....
2
votes
1answer
72 views

Under what conditions does $H(X\mid f(Y))=H(X\mid Y)$?

I have the problem that I cannot solve: Under what conditions does $H(X∣f(Y))=H(X∣Y)$? I would like to draw a result about the relation between $p_X(\cdot | g(Y))$ and $p_X(\cdot | Y)$. Are they equal?...