# Questions tagged [entropy]

This tag is for questions about mathematical entropy. If you have a question about thermodynamical entropy, visit Physics Stack Exchange or Chemistry Stack Exchange instead.

1,412 questions
Filter by
Sorted by
Tagged with
13 views

### Mutual Information between $v_1$ and $v_2$ coming from the same Inverse-Wishart distribution?

Say that $\left(\begin{matrix} v_1 & c\\ c & v_2 \end{matrix}\right)$ is a bivariate covariance matrix that comes from an Inverse-Wishart distribution $W^{-1}(\Psi, \nu)$. Then what is the ...
29 views

### Inequality relating entropy to mutual information

Let $\{X_n\}$ be a sequence of independent, discrete random variables, and let $Z$ be another discrete random variable. Show that $$H(Z)\geq\sum_{i=1}^\infty I(X_i;Z)$$ where $H$ is the entropy and $I$...
29 views

### Understanding Rokhlin's theorem of cross-sections

I am reading the book Conformal Fractals: Ergodic Theory Methods by Przytycki and Urbański (the book is available legally on a site of the first author https://www.impan.pl/~feliksp/ksiazka1.pdf ) ...
1 vote
57 views

7 views

### Any reference related to regulating the variation of entropies?

I need some reference papers related to my problem. I have estimations as N normal distributions, but their variance tends to 0. It's because distributions are aggregated to one normal whose variance ...
1 vote
46 views

### Understanding Uhlmann Monotonicity Theorem on von Neumann Algebras

This is my first post, so apologies if this is a bad post. I'm reading "Quantum Entropy and its use' by M. Ohya and D. Petz. Theorem 5.3 states Let $M_1$ and $M_2$ be von Neumann algebras with ...
1 vote
16 views

### Bound KL divergence between two distributions by KL divergence of two Gaussian mixture models

I'm trying to bound the KL divergence between two continuous random variables with the KL divergence between two Gaussian mixture approximations motivated by the fact that the Gaussian mixture model ...
1 vote
45 views

### Chemical Entropy vs. Mathematical Entropy

In high school physics and chemistry classes, we were told that entropy is a measure of disorder in a physical system. For example, molecules that are relatively stationary correspond to a lower ...
1 vote
36 views

Recently, I have learned about the principle of Maximum Entropy with regards to Probability Distribution (https://www.youtube.com/watch?v=2gTrsLVnp9c) - in particular, when certain "information&...
147 views

### Does "Entropy" explain why the Normal Distribution is so "Popular"?

Recently, I have learned about the Principle of Maximum Entropy with regards to Probability Distribution - in particular, when certain "information" (i.e. constraints) is available about ...
1 vote
21 views

### Computing entropies of designs.

I've been told to assume $6$ coins are weighed on a chemical balance (two-pan) scale. We are told exactly two coins are fake and that fake coins are heavier than real ones. I've been given two designs ...
8 views

21 views

### Entropy of convex combination of dirac points are positive

Let $X$ be a compact metric space. $T:X \to X$ be a homeomorphism. Assume that the measure$\mu=\lambda \delta_{a}+(1-\lambda)\delta_{b},$ where $\delta$ is the Dirac measure and $0<\lambda<1.$ ...
20 views

### How to get Information Entropy and Chi-Square Test value for an encrypted image

I was reading this paper. Under the section 4.3 and 4.5 they add The Chi-Square Test Analysis of Cipher Image and Information ...
103 views

### What is the role of the logarithm in Shannon's entropy?

I am a layman interested in understanding why the foundation of Shannon's entropy is logarithmic. To that end I've read the answers here, at the Cross Validated Stack, but I'm not technical enough to ...
1 vote
34 views

38 views

### Is it possible for a programme to return an output with more entropy than the entropy of the programme's code itself?

Suppose that $P$ is a random variable that takes values in the space of computer programmes. So, basically, $P$ is some code (e.g. C, Python, or some theoretical one like the one used in the tapes of ...
1 vote
74 views

29 views

### Mutual Information and Entropy calculation

It is well known that Shannon's joint entropy ($H(X,Y)$) as well as mutual information ($I(X;Y)$) between two variables $X$ and $Y$ are non-negative based on Jensen's inequality. I read in a source ...
23 views

Consider a random variable $X$ taking values in $\{0,1,\ldots,n\}$ and $Y$ takes values in $\{0,1\}$. Let $a_{i}=P\left(X={i}\right), b_{j}=P(Y=j)$. Also, $p_{i}=P\left(Y=0 \mid X=x_{i}\right), q_{i}=... 1 vote 0 answers 36 views ### In information entropy, how do nats relate to any representation of states? Calculating the information entropy depends on taking the logarithms of probabilities in some base. If I use base 2, then the entropy is in "bits". The measure of bits is close to the ... 0 votes 0 answers 14 views ### Solving the Flory-Huggins counting problem when the polymers and solvent have colors I am trying to construct a Flory-Huggins type lattice for a polymer and solvent with "colors". Essentially, each monomer segment and solvent segment has a color associated with it, and beads ... 0 votes 0 answers 23 views ### Chi-square and Taylor expansion of relative entropy I'm trying to show a relation between relative entropy and Chi-square, more specifically,$\chi^2 = \sum_{x}\frac{(p(x) - q(x))^2}{q(x)}$is twice the first term in the taylor series expansion of$D(p||...
If $X$ is a well-behaved continuous random variable, is it true that $$H(XX) = H(X)$$ $$I(X:X) = H(X)$$ This is certainly true for discrete variables, since (assuming X = Y) H(XY) = H(X|Y) + H(Y) = ...