# Questions tagged [entropy]

This tag is for questions about mathematical entropy. If you have a question about thermodynamical entropy, visit Physics Stack Exchange or Chemistry Stack Exchange instead.

1,552 questions
Filter by
Sorted by
Tagged with
17 views

### Shannon entropy property

I need to prove this inequality $S(A|B)\leq S(A)$ being S the Shannon entropy $S(B)=-\sum_{\beta}p(\beta)ln(p(\beta))$, and $S(A|B)=-\sum_{\alpha\beta}p(\beta)p(\alpha|\beta)ln(p(\alpha|\beta))$ using ...
1 vote
31 views

### Inequailities on conditional entropy of a markov chain

Let $x_{n}$ a stationary discrete markov chain . I have to prove that: For $n\geq 1$ $$H(X_{n} | X_{0}) \geq H(X_{n-1} | X_{0})$$ $$H(X_{0} | X_{n}) \geq H(X_{0} | X_{n-1})$$ Where $H(X)$ is the ...
9 views

### Find maximum entropy dirichlet distribution with given mean

Given a categorical distribution p, any Dirichlet distribution Dir(k*p) will have mean p for ...
26 views

### Entropy increase of a random walk on the sphere

Let $X_0$ be some vector on the unit sphere of $\mathbb{R}^n$ and let $\rho_0$ be the distribution of its entries, $$\rho_0(z) = \frac{1}{n} \sum_i \delta\left(z-X_0^{i}\right)$$ Assume that $n$ is ...
154 views

### How to prove $(p\log p+q\log q)^2\leq -\log(p^2+q^2)\log 2$?

I came up with the following conjecture while tacking another problem: Conjecture. Let $p \in [0, 1]$ and $q = 1-p$. Then $$(p \log p + q \log q)^2 \leq -\log(p^2 + q^2)\log 2$$ A numerical ...
1 vote
33 views

### Generating Sequences with Monotonically Increasing Entropy

Context: Consider an ensemble $X = (x; A; P)$. Let's examine $X^N$, which represents the set of all possible strings of length $N$produced by $X$. The objective is to systematically derive sequences ...
39 views

### Is Entropy of a Discrete Random Variable Condition on a Continuous Random Variable Always bigger than or Equal to Zero? [closed]

Let $X$ be a discrete random variable and $Y$ be a continuous random variable. Is the conditional entropy of $X$ given $Y$ always positive, i.e., $H(X|Y)\ge0$?
1 vote
222 views

### The correct way to integrate the variational free-energy formula

Lets define the following probability density distributions as: \begin{align} p(θ) &= N(θ; 0,1) \\ q(θ) &= N(θ; μ,σ^2)\\ p(y|θ,x) &= N(y; θx,σ_n^2) \end{align} where $N(x; m,v)$ ...
1 vote
24 views

### Log-Sobolev on an incomplete manifold

Maybe my question is a nonsense but I prefer to ask. Take $M$ a manifold that is not complete (ie the exponential is not defined for all values of $TM$) and $V$ a potential defined on such manifold. ...
16 views

1 vote
22 views

### Variance of Value in Monte Carlo Estimation of Shannon Entropy.

The Shannon entropy of a source $X$ (in bits) is defined as:- $$H(X) = -\sum_{x \in X} {p(x) \log_2 p(x)}$$ Following principles in Variance of Area and Average Estimators in Monte Carlo ...
52 views

### Is the Rényi entropy a continuous function with respect to the parameter $\alpha$?

The Rényi entropy of order $\alpha$, where $\alpha > 0$ and $\alpha \neq 1$, is defined as $$\mathrm{H}_\alpha(X)=\frac{1}{1-\alpha} \log \left(\sum_{i=1}^n p_i^\alpha\right)$$ Here, $X$ is a ...
25 views

### Does the conditional entropy H(X | (Y,Z)) equal H((X | Y) | Z)?

X,Y,Z are random variables of course. I have seen that this is considered equal sometimes but I need confirmation. I tried using H(A,B) = H(A|B) + H(B): H(X | (Y,Z)) = H(X,Y,Z) - H(Y,Z) and H(Y,Z) = H(...
21 views

1 vote
32 views

### Total Correlation is difference of relative entropies in general?

Motivation Consider a finite set $[q]=\{1,\dots,q\}$, random variables $X_1,\dots,X_k\in[q]$, and their product $X=X_1\otimes\cdots\otimes X_k\in[q]^k$, i.e. the components of $X$ are independent. Let ...
80 views

### Intuitive interpretation of entropy

I'm trying to understand entropy and KL divergence. While it makes sense in a simplistic case, such as the case of a coin flip, I struggle wrapping my head around it when it is a more complicated case ...
84 views

75 views

### Minimizing Differential Entropy of a Gaussian Random Variable Conditioned on Sum of Gaussian and Non-Gaussian Random Variables

Let $A$ be a normal distribution with variance $\sigma_A^2$ and $B$ be a continuous random variable with variance $\sigma_B^2$. Here, $A$ and $B$ are independent. Is the Gaussian distribution for $B$ ...
1 vote