Questions tagged [entropy]

This tag is for questions about mathematical entropy. If you have a question about thermodynamical entropy, visit Physics Stack Exchange or Chemistry Stack Exchange instead.

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

Mutual Information between $v_1$ and $v_2$ coming from the same Inverse-Wishart distribution?

Say that $\left(\begin{matrix} v_1 & c\\ c & v_2 \end{matrix}\right)$ is a bivariate covariance matrix that comes from an Inverse-Wishart distribution $W^{-1}(\Psi, \nu)$. Then what is the ...
user avatar
  • 1,612
2 votes
1 answer
29 views

Inequality relating entropy to mutual information

Let $\{X_n\}$ be a sequence of independent, discrete random variables, and let $Z$ be another discrete random variable. Show that $$H(Z)\geq\sum_{i=1}^\infty I(X_i;Z)$$ where $H$ is the entropy and $I$...
user avatar
2 votes
1 answer
29 views

Understanding Rokhlin's theorem of cross-sections

I am reading the book Conformal Fractals: Ergodic Theory Methods by Przytycki and Urbański (the book is available legally on a site of the first author https://www.impan.pl/~feliksp/ksiazka1.pdf ) ...
user avatar
1 vote
1 answer
57 views

Fano's Inequality without conditioning

Suppose $X\sim P$ is a random variable taking values on an alphabet $\mathcal{A}=\{1,\dots,m\}$, such that $p:=P(1)>P(k)$ for $k\neq1$. The minimum-probability-of-error predictor of $X$ is $\hat{X}=...
user avatar
1 vote
0 answers
32 views

Example of random variable with infinite entropy

Let $X\sim P$ on $A=\{2,3,\dots\}$, where $P(k)=\frac{C}{k(\log k)^2}$ for $k\geq2$ with $C$ some normalising constant. Show that $H(X)=\infty$. My attempt so far: I have shown, by direct computation ...
user avatar
0 votes
0 answers
19 views

Confused about notation for conditional mutual information

Consider discrete random variables $Z$, $W$, $X$, and some event $\mathcal{E}$: I'm confused about the meaning of the conditional mutual information $I[ Z : W \mid X, \mathcal{E} ]$. I'm aware of the ...
user avatar
  • 29
0 votes
0 answers
17 views

Understanding problem of chain Rule for mutial information

So I have to proof $ I(X;Z|Y) = I(Z;Y|X) - I(Z;Y) + I(X;Z) $ Write mutual information in terms of entropy or use the chain rule for mutual information for an immediate proof. So I want to directly ...
user avatar
  • 103
0 votes
1 answer
34 views

Entropy: Show $ H(X|Y) + H(Y|Z) \geq H(X|Z) $

I got following exercise: Let $X$; $Y$; $Z$ be random variables. Show that $H(X \mid Y) + H(Y \mid Z) \geq H(X \mid Z)$. Hint: consider $H(X, Y \mid Z)$. We showed in the lecture that $H(X,Y \mid Z) = ...
user avatar
  • 103
0 votes
0 answers
7 views

Any reference related to regulating the variation of entropies?

I need some reference papers related to my problem. I have estimations as N normal distributions, but their variance tends to 0. It's because distributions are aggregated to one normal whose variance ...
user avatar
1 vote
0 answers
46 views

Understanding Uhlmann Monotonicity Theorem on von Neumann Algebras

This is my first post, so apologies if this is a bad post. I'm reading "Quantum Entropy and its use' by M. Ohya and D. Petz. Theorem 5.3 states Let $M_1$ and $M_2$ be von Neumann algebras with ...
user avatar
1 vote
0 answers
16 views

Bound KL divergence between two distributions by KL divergence of two Gaussian mixture models

I'm trying to bound the KL divergence between two continuous random variables with the KL divergence between two Gaussian mixture approximations motivated by the fact that the Gaussian mixture model ...
user avatar
1 vote
0 answers
45 views

Chemical Entropy vs. Mathematical Entropy

In high school physics and chemistry classes, we were told that entropy is a measure of disorder in a physical system. For example, molecules that are relatively stationary correspond to a lower ...
user avatar
1 vote
0 answers
36 views

Why is "More Information" better than "Less Information"?

Recently, I have learned about the principle of Maximum Entropy with regards to Probability Distribution (https://www.youtube.com/watch?v=2gTrsLVnp9c) - in particular, when certain "information&...
user avatar
  • 1,698
7 votes
1 answer
147 views

Does "Entropy" explain why the Normal Distribution is so "Popular"?

Recently, I have learned about the Principle of Maximum Entropy with regards to Probability Distribution - in particular, when certain "information" (i.e. constraints) is available about ...
user avatar
  • 1,698
1 vote
1 answer
21 views

Computing entropies of designs.

I've been told to assume $6$ coins are weighed on a chemical balance (two-pan) scale. We are told exactly two coins are fake and that fake coins are heavier than real ones. I've been given two designs ...
user avatar
  • 23
0 votes
0 answers
8 views

Existence of measure within KL and Levy-Prokhorov distance

Let $p_1,q_1,p_2$ be probability measures. Does there exist a measure $q_2$ such that $$ KL(q_2||p_2) \leq KL(q_1||p_1) \quad \text{and} \quad \pi_{LP}(q_1,q_2) \leq \pi_{LP}(p_1,p_2) $$ where $KL(\...
user avatar
  • 45
3 votes
1 answer
77 views

Relation between the entropies $h(\sum w_i f_i)$, $h(\sum w_i g_i)$, when $h(f_i) \ge h(g_i)$?

Problem $f_i, g_i$ are the probability density functions of the symmetric, unimodal distributions with the common center $c_i$. Assume the following: All $g_i$ are the translations of $g_0$ (whose ...
user avatar
  • 1,934
1 vote
1 answer
29 views

Derivation of the Conditional Maximum Entropy distribution

I am trying to derive the conditional maximum entropy distribution in the discrete case, subject to marginal and conditional empirical moments. We assume that we have access to the empirical moments, $...
user avatar
  • 532
0 votes
0 answers
21 views

Entropy of convex combination of dirac points are positive

Let $X$ be a compact metric space. $T:X \to X$ be a homeomorphism. Assume that the measure$\mu=\lambda \delta_{a}+(1-\lambda)\delta_{b},$ where $\delta$ is the Dirac measure and $0<\lambda<1.$ ...
user avatar
  • 516
0 votes
0 answers
20 views

How to get Information Entropy and Chi-Square Test value for an encrypted image

I was reading this paper. Under the section 4.3 and 4.5 they add The Chi-Square Test Analysis of Cipher Image and Information ...
user avatar
4 votes
2 answers
103 views

What is the role of the logarithm in Shannon's entropy?

I am a layman interested in understanding why the foundation of Shannon's entropy is logarithmic. To that end I've read the answers here, at the Cross Validated Stack, but I'm not technical enough to ...
user avatar
1 vote
1 answer
34 views

Prove that the Werner-Holevo channel $C(X)\equiv\frac1{n\pm1}({\rm Tr}(X)I_n\pm X^T)$ is a quantum channel

Kind of a nooby question. But how would you actually go about proving that the Werner Holevo channel is a quantum channel? $$ C(X) = \frac{1}{n \pm 1} \left(\text{Tr}(X)\mathbb{I}_n \pm X^{\top} \...
user avatar
1 vote
1 answer
69 views

What is special about the maps whose measure-theoretic entropy is equal to zero?

When I calculate the measure-theoretic entropy of different maps, I found that some of them is equal to zero, so I am curious about what is special about the map whose entropy is zero? Does the ...
user avatar
  • 45
0 votes
1 answer
21 views

Combining Shannon Entropy with a notion of bit-rate

If you have a sequence of samples from a finite alphabet where the $i$th symbol has probability $p_i$ the shannon entropy of each symbol $H = \sum_i p_i \log_2(p_i)$. That is each symbol carries $H$ ...
user avatar
  • 1,900
3 votes
1 answer
59 views

Generator of doubling map when calculating its Measure-theoretic entropy

I saw an example to show how to calculate the Measure-theoretic entropy, and here is the example: Let $T:X\to X$ be the doubling map $T(x)=2x\;(mod\;1)$ and there is a partition $\alpha=\{[0,\frac{1}{...
user avatar
  • 45
0 votes
1 answer
12 views

How much entropy exists in knowing the specific sorting of a list of $n$ many items with repetition in the list?

If I have a bag of $b$ many balls, each numbered from $1, 2, \ldots, b$, and I uniformly-randomly pick one ball. Then I ask you "how much information would you gain should I tell you the ball ...
user avatar
  • 385
0 votes
0 answers
30 views

Mutual Information of Vectors with Large Inner Product

If we have a joint distribution of two (complex) vectors $x,y\in \mathbb{C}^d$ of norm $1$ such that their inner product $\langle x|y\rangle$ is $1-\epsilon$, can we lower bound the mutual information ...
user avatar
  • 1
0 votes
0 answers
15 views

Entropy rate of a sub-set depending on sampling probability

Assuming we have a source producing stochastic binary sequence (0 & 1) with temporal correlation. With long enough observation of the sequence, we can calculate the entropy rate of the source H-...
user avatar
  • 1
7 votes
2 answers
280 views

Prove that $f(x,y)+f(y,z)\ge f(x,z)$ where $f(x,y)=\sqrt{x\ln x+y\ln y-(x+y)\ln(\frac{x+y}2)}$

Denote $f(x,y)=\sqrt{x\ln x+y\ln y-(x+y)\ln(\frac{x+y}2)}$. Show that $f(x,y)+f(y,z)\ge f(x,z)$ for $x,y,z> 0$. This is a question from a friend, which is a deep learning homework. It looks like ...
user avatar
  • 2,536
10 votes
2 answers
389 views

Entropy of fair but correlated coin flips

Consider the joint distribution, $p(\xi_1,...\xi_N)$, with components defined as $\xi_i=\mathrm{sign}(x_i)$, with $(x_1,...,x_N)\sim\mathcal{N}(0,\Sigma)$ with $ \Sigma_{ij}=\delta_{ij}+(1-\delta_{ij})...
user avatar
  • 165
0 votes
0 answers
28 views

Mutual information as supermodular function under independence?

Let $Y$ be a response variable and $F$ be a set of features such that $f,x \in F$ and $S \subset F$. I am interested in the difference of mutual informations and would like to show the following ...
user avatar
  • 195
0 votes
0 answers
11 views

Mapping number into number with high algorithmic entropy

Is there a function that takes some n-bit number with some Kolmogorov complexity and transform it into n-bit number with high Kolmogorov complexity? Such a function of course can't be bijective, ...
user avatar
  • 149
0 votes
0 answers
37 views

Cross entropy and log-likelihood

I think it's pretty clear to me that average log-likelihood is equivalent to negative cross-entropy for discrete distributions, as shown here: $$\frac{1}{N}\log\mathcal{L}(\theta) = \frac{1}{N}\log \...
user avatar
0 votes
1 answer
38 views

Is it possible for a programme to return an output with more entropy than the entropy of the programme's code itself?

Suppose that $P$ is a random variable that takes values in the space of computer programmes. So, basically, $P$ is some code (e.g. C, Python, or some theoretical one like the one used in the tapes of ...
user avatar
  • 385
1 vote
2 answers
74 views

What's the intuition behind typical sequences?

Given a probability distribution $x\mapsto p(x)$, some integer $n>0$, and some $\epsilon>0$, one defines $\epsilon$-typical sequences as those sequences $\boldsymbol{x}\equiv(x_1,...,x_n)$ $$\...
user avatar
  • 5,523
0 votes
2 answers
69 views

Why does $H(X|Y)$ equal the "missing information" of $Y$ about $X$?

I've seen mentioned in (Horodecki, Oppenheim, Winter 2005) the fact that the conditional information equals the amount of information that Alice needs to send Bob in order for him to fully reconstruct ...
user avatar
  • 5,523
0 votes
0 answers
21 views

Maximal entropy function of multiple random variables

Assume that we have a known multivariate distribution of $N$ variables denoted by $f(\vec{X})$. Consider the random variable $Y = g(\vec{X})$ defined by a smooth function $g:\mathbb{R^{n}}\rightarrow\...
user avatar
0 votes
1 answer
31 views

Calculation for the entropy of binomial random graph

In my classes, we found that for a graph $\mathbf{a}\in G(n,p)$, where we have labelled the graphs by their adjacency matrices, $P(\mathbf{a}) = \prod _{i<j}p^{a_{ij}}(1-p)^{(1-a_{ij})}$ We define ...
user avatar
  • 2,947
5 votes
2 answers
71 views

Entropy and probabilistic Algorithms

Recall entropy, from basic information theory: The entropy of a probability distribution $D$ on a finite set $X$ is $$H(D)=\sum_{x\in X}{p(x) \cdot \log_2{\!(1/p(x))}}$$ I was able to prove that the ...
user avatar
0 votes
0 answers
22 views

Entropy change for random probability distributions

Suppose we are dealing with a stochastic process $(X_n)_{n \geq 1}$, where for each $n$, the prediction rule for $X_{n+1}$ is induced by $X_1,\ldots,X_n$. To give some context, in my case that would ...
user avatar
3 votes
0 answers
57 views

When is the Kullback–Leibler divergence continuous?

For two probability measures $P,Q$, if $P\ll Q$, then the Kullback–Leibler divergence (or the relative entropy) is defined as: $$D_{KL}(P||Q)=\int\log\bigg(\frac{dP}{dQ}\bigg)dP.$$ It is well known ...
user avatar
  • 298
0 votes
0 answers
27 views

Estimate entropy from samples under multivariate Bernoulli distribution

Could you find any way to estimate entropy from samples under multivariate Bernoulli distribution? Formally, for a multivariate Bernoulli distribution, getting its joint probability is somehow not ...
user avatar
0 votes
0 answers
11 views

proof) a specific condition on maximum of a sum of weighted log(Xi)

Let Y $$Y=\sum_{i=1}^n \alpha_ilog X_i$$ $$(\alpha_i> 0, X_i > 0)$$ to maximize Y with respect to Xi, put Xi as below (except for that all Xi are the same) $$\frac{X_i}{\sum_{i=1}^nX_i}=\frac{\...
user avatar
0 votes
1 answer
29 views

Mutual Information and Entropy calculation

It is well known that Shannon's joint entropy ($H(X,Y)$) as well as mutual information ($I(X;Y)$) between two variables $X$ and $Y$ are non-negative based on Jensen's inequality. I read in a source ...
user avatar
0 votes
0 answers
23 views

comparing mutual information under different constraints

Consider a random variable $X$ taking values in $\{0,1,\ldots,n\}$ and $Y$ takes values in $\{0,1\}$. Let $a_{i}=P\left(X={i}\right), b_{j}=P(Y=j)$. Also, $ p_{i}=P\left(Y=0 \mid X=x_{i}\right), q_{i}=...
user avatar
1 vote
0 answers
36 views

In information entropy, how do nats relate to any representation of states?

Calculating the information entropy depends on taking the logarithms of probabilities in some base. If I use base 2, then the entropy is in "bits". The measure of bits is close to the ...
user avatar
  • 371
0 votes
0 answers
14 views

Solving the Flory-Huggins counting problem when the polymers and solvent have colors

I am trying to construct a Flory-Huggins type lattice for a polymer and solvent with "colors". Essentially, each monomer segment and solvent segment has a color associated with it, and beads ...
user avatar
  • 255
0 votes
0 answers
23 views

Chi-square and Taylor expansion of relative entropy

I'm trying to show a relation between relative entropy and Chi-square, more specifically,$\chi^2 = \sum_{x}\frac{(p(x) - q(x))^2}{q(x)}$ is twice the first term in the taylor series expansion of $D(p||...
user avatar
  • 111
0 votes
0 answers
13 views

Is my intuition about the differential entropy and mutual information correct

If $X$ is a well-behaved continuous random variable, is it true that $$H(XX) = H(X)$$ $$I(X:X) = H(X)$$ This is certainly true for discrete variables, since (assuming X = Y) $$H(XY) = H(X|Y) + H(Y) = ...
user avatar
3 votes
1 answer
47 views

Is Dirichlet energy related with entropy?

Intuitively, I feel that Dirichlet energy is related with entropy. And entropy seems to be equivalent with some discrete form of Dirichlet energy. Is this a nice intuition? Is there something worth ...
user avatar

1
2 3 4 5
29