Questions tagged [entropy]
This tag is for questions about mathematical entropy. If you have a question about thermodynamical entropy, visit Physics Stack Exchange or Chemistry Stack Exchange instead.
1,552
questions
0
votes
1
answer
17
views
Shannon entropy property
I need to prove this inequality $S(A|B)\leq S(A)$ being S the Shannon entropy $S(B)=-\sum_{\beta}p(\beta)ln(p(\beta))$, and $S(A|B)=-\sum_{\alpha\beta}p(\beta)p(\alpha|\beta)ln(p(\alpha|\beta))$ using ...
- 103
1
vote
1
answer
31
views
Inequailities on conditional entropy of a markov chain
Let $x_{n}$ a stationary discrete markov chain . I have to prove that:
For $n\geq 1$
$$ H(X_{n} | X_{0}) \geq H(X_{n-1} | X_{0})$$
$$ H(X_{0} | X_{n}) \geq H(X_{0} | X_{n-1})$$
Where $H(X)$ is the ...
- 109
0
votes
0
answers
9
views
Find maximum entropy dirichlet distribution with given mean
Given a categorical distribution p, any Dirichlet distribution Dir(k*p) will have mean p for ...
- 15
0
votes
0
answers
26
views
Entropy increase of a random walk on the sphere
Let $X_0$ be some vector on the unit sphere of $\mathbb{R}^n$ and let $\rho_0$ be the distribution of its entries,
$$
\rho_0(z) = \frac{1}{n} \sum_i \delta\left(z-X_0^{i}\right)
$$
Assume that $n$ is ...
4
votes
1
answer
154
views
How to prove $(p\log p+q\log q)^2\leq -\log(p^2+q^2)\log 2$?
I came up with the following conjecture while tacking another problem:
Conjecture. Let $p \in [0, 1]$ and $q = 1-p$. Then
$$ (p \log p + q \log q)^2 \leq -\log(p^2 + q^2)\log 2 $$
A numerical ...
- 156k
1
vote
0
answers
33
views
Generating Sequences with Monotonically Increasing Entropy
Context:
Consider an ensemble $X = (x; A; P)$. Let's examine $X^N$, which represents the set of all possible strings of length $N $produced by $X$. The objective is to systematically derive sequences ...
- 111
0
votes
1
answer
39
views
Is Entropy of a Discrete Random Variable Condition on a Continuous Random Variable Always bigger than or Equal to Zero? [closed]
Let $X$ be a discrete random variable and $Y$ be a continuous random variable. Is the conditional entropy of $X$ given $Y$ always positive, i.e., $H(X|Y)\ge0$?
- 81
1
vote
1
answer
222
views
The correct way to integrate the variational free-energy formula
Lets define the following probability density distributions as:
$$
\begin{align}
p(θ) &= N(θ; 0,1) \\
q(θ) &= N(θ; μ,σ^2)\\
p(y|θ,x) &= N(y; θx,σ_n^2)
\end{align}
$$
where $N(x; m,v)$ ...
- 214
1
vote
0
answers
24
views
Log-Sobolev on an incomplete manifold
Maybe my question is a nonsense but I prefer to ask.
Take $M$ a manifold that is not complete (ie the exponential is not defined for all
values of $TM$) and $V$ a potential defined on such manifold.
...
- 41
0
votes
1
answer
16
views
Recovering Entropy Rate Property for Random Walk Across Unweighted Graph
I am self-studying Information Theory and came across this problem concerning the entropy rate of a random walk across this graph. For all logarithms, I am working in base 2.
$$\mu=(3/16, 3/16, 3/16, ...
1
vote
1
answer
66
views
$\min$-entropy for the uniform distribution on $[𝑛]$
The min-entropy of a distribution $\nu$ on $[n]$ is given as:
$$H_{\infty}(\nu)=\min_{i} \log(\frac{1}{\nu(i)})$$
Now we will prove that that for every distribution $\nu$ on $[n]$ and for $U$ being ...
- 648
0
votes
0
answers
9
views
Opitimization with softmax function
I am trying to solve the following problem.
Given a $n$-dimension column vector $\mathbf z$ with each element $z_i\in\mathbb R$, I wish to minimize the following objective function.
$\mathbf z^T\cdot ...
0
votes
0
answers
9
views
Help with a Riemann-type limit involving a function of the differential
$X$ is a continuous random variable and $F_X(x)$ is its differentiable cumulative density function. I'm considering what happens if I discretize $X$ to intervals of length $\Delta x$ (i.e. a discrete $...
2
votes
0
answers
32
views
Maximum entropy for continuous distributions, optimization problem
Reading E. T. Jaynes book. In chapter 12 he introduces an extension to Shannon's entropy for continuous distributions:
(*) $H_{I}^{c} = - \int \text{d}z \, p(z|I) \log{\frac{p(z|I)}{m(z)}}$
which has ...
- 21
0
votes
0
answers
30
views
Question from Katok and Hasselblat (Exercise 4.3.1)
I was trying to solve question 4.3.1 from Katok and Hasselblat but I couldn't make my argument complete. Could you look at it?
This is the question.
Let $\xi, \eta \in \mathcal{P}_m$. Prove that for ...
- 963
0
votes
0
answers
26
views
Minimum mean length of code words
I need help with this task, so if anyone is willing to help me, I would be grateful.
The task is:
Given a discrete information source that generates symbols from the set ...
- 457
0
votes
1
answer
36
views
Why is $q$-ary entropy defined as such?
The q-ary entropy for any $q\in\mathbb{N}$ and $q\geq 2$ and $x\in[0,1]$ is defined as
$$H_q(x)=x \log _q(q-1)-x \log _q(x)-(1-x) \log _q(1-x) .$$
One recovers the binary entropy with $q=2$. What is ...
- 1,632
1
vote
0
answers
37
views
What is the relation between off-diagonal entries and the determinant of a SPSD matrix?
I am interested in the differential entropy of a multivariate Gaussian distribution, $h(f)$. In particular, I want to know how does $h(f)$ change with changes in the off-diagonal elements of the ...
- 686
0
votes
3
answers
53
views
Finding the bounds of the missing values of a symmetric positive semidefinite matrix
Suppose we have a symmetric matrix that we know is positive semi-definite and has missing entries. For example:
$
\begin{pmatrix}
1&1&1\\
1&1&x\\
1&x&1
\end{pmatrix}
$
How can ...
- 686
1
vote
0
answers
36
views
A question about the set of probability vectors
Let
\begin{equation}
P= \{(x_i)_{i\in\mathbb{N}}: x_i\geq 0, \sum_{i=1}^\infty x_i=1\}
\end{equation}
Every $p\in P$ is called probability vector.
The following notation is introduced in a book, but ...
- 1,233
0
votes
3
answers
57
views
Maximum entropy distribution via Lagrange multiplier
I am following this tutorial, where we derive common probability distributions based on a constraint and the entropy equation for a discrete random variable.
I am stuck on a step: it's labeled step 8 ...
0
votes
0
answers
27
views
Independences implied by the Maximum Entropy distribution, subject to dependency constraints
Consider a discrete distribution $P(X,Y,Z)$. If we wish to find the maximum entropy distribution (MaxEntDist) of P subject to maintaining the marginals $P(X), P(Y)$ and $P(Z)$, it is given by $P_{X:Y:...
1
vote
1
answer
50
views
Entropy of a Random Variable - Sending Message about Outcome
Consider a random variable with probabilities taking on the values $1,2,\ldots,8$ with probabilities given by $$\left(\frac{1}{2}, \frac{1}{4}, \frac{1}{8}, \frac{1}{16}, \frac{1}{64}, \frac{1}{64}, \...
- 725
1
vote
0
answers
22
views
Variance of Value in Monte Carlo Estimation of Shannon Entropy.
The Shannon entropy of a source $X$ (in bits) is defined as:-
$$ H(X) = -\sum_{x \in X} {p(x) \log_2 p(x)} $$
Following principles in Variance of Area and Average Estimators in Monte Carlo ...
- 458
4
votes
2
answers
52
views
Is the Rényi entropy a continuous function with respect to the parameter $\alpha$?
The Rényi entropy of order $\alpha$, where $\alpha > 0$ and $\alpha \neq 1$, is defined as
$$
\mathrm{H}_\alpha(X)=\frac{1}{1-\alpha} \log \left(\sum_{i=1}^n p_i^\alpha\right)
$$
Here, $X$ is a ...
- 7,753
2
votes
0
answers
25
views
Does the conditional entropy H(X | (Y,Z)) equal H((X | Y) | Z)?
X,Y,Z are random variables of course.
I have seen that this is considered equal sometimes but I need confirmation.
I tried using H(A,B) = H(A|B) + H(B):
H(X | (Y,Z)) = H(X,Y,Z) - H(Y,Z)
and
H(Y,Z) = H(...
- 31
0
votes
0
answers
21
views
Reference: maximal entropy of a random subset with a fixed mean
I am looking for a reference (e.g. in some textbook) of the following:
Let $F$ be a finite set, and $X$ a random subset of $F$ (i.e., a random variable with values in the power set of $F$) such that $\...
- 1
2
votes
0
answers
39
views
Entropy of a Bernoulli random variable with logit probability.
I need to compute the expected entropy of a Bernoulli random variable whose probability of success is given by $\frac{1}{1+\exp(-x)}$ where $x$ is normally distributed. This leaves me stuck with ...
- 85
2
votes
2
answers
71
views
Probability that a sequence is typical with respect to some distribution
Suppose that sequence $Z_1,Z_2,\dots,Z_n$ is drawn i.i.d. from some distribution (pmf) $Q_Z$. Prove that the probability that it's typical with respect to some other distribution $P_Z$ is roughly $2^{...
- 4,481
1
vote
1
answer
32
views
Total Correlation is difference of relative entropies in general?
Motivation
Consider a finite set $[q]=\{1,\dots,q\}$, random variables $X_1,\dots,X_k\in[q]$, and their product $X=X_1\otimes\cdots\otimes X_k\in[q]^k$, i.e. the components of $X$ are independent.
Let ...
- 3,184
3
votes
1
answer
80
views
Intuitive interpretation of entropy
I'm trying to understand entropy and KL divergence. While it makes sense in a simplistic case, such as the case of a coin flip, I struggle wrapping my head around it when it is a more complicated case ...
- 143
2
votes
1
answer
84
views
Lossy entropy coding
Using arithmetic coding we can encode a sequence of independent biased coin flips using $<1$ bit per coin flip. For example consider a coin with $p=0.8$. We can encode the flips of this coin using $...
- 201
1
vote
1
answer
42
views
Example of finite closed cover with entropy strictly greater than topological entropy
I'm reading "Topological entropy bounds measure-theorettic entropy", by L.W. Goodwyn. After Proposition 2, he mentions that "finite closed cover can yield entropy strictly greater than ...
- 41
3
votes
0
answers
58
views
Prove that E(t) is increasing in t
Consider an irreducible continuous-time Markov chain $X_t$ on finite state space with $Q$-matrix $Q=(q_{ij}, i,j=1,2,\cdots, N)$, in which $q_{ij}=q_{ji}$. Fix initial state $j$, let $P_i(t)$ be the ...
- 125
0
votes
1
answer
48
views
How do we describe my definition of non-uniformity in a 2-d space mathematically?
I'm having trouble desribing the following:
Suppose, using the Lebesgue Outer measure, we define a function $f:[0,1]\to[0,1]$ that is measurable in the sense of Caratheodory.
I want to make the ...
- 1
0
votes
0
answers
23
views
Proof of differetial entropy of the Multivartiate Gaussian
How do you get from the left hand side to the right hand side of this equation, this is converting the multivariate gaussian in to differential entropy.
$\mathbb{E}[(x-\mu)^T\Sigma^{-1}(x-\mu)]=\...
- 21
2
votes
1
answer
75
views
Minimizing Differential Entropy of a Gaussian Random Variable Conditioned on Sum of Gaussian and Non-Gaussian Random Variables
Let $A$ be a normal distribution with variance $\sigma_A^2$ and $B$ be a continuous random variable with variance $\sigma_B^2$. Here, $A$ and $B$ are independent. Is the Gaussian distribution for $B$ ...
- 81
1
vote
1
answer
42
views
Entropy of a random vector: 3 coins
I have a sample question for an exam and would like some help understanding if I'm approaching it correctly. The question:
...
- 55
1
vote
2
answers
71
views
Shuffle poker deck increase entropy but function decrease entropy
Let $X$ be a random variable, $f(X)$ is a function.
And we have
$$
\begin{align}
I(X;f(X)) &= H(X)-H(X|f(X)) \\
&= H(f(X))-\underbrace{H(f(X)|X)}_{=0}\\
H(f(X)) &= H(X)-H(X|f(X))\leq H(X)
\...
- 41
0
votes
1
answer
27
views
Are there maximum entropy distributions with fixed moments of a given order?
A characterization of the multivariate Gaussian distribution with a fixed mean and covariance matrix is that it is the unique probability distribution with fixed mean and covariance matrix that ...
- 395
0
votes
1
answer
31
views
Finding the differential entropy for random variable
I'm having trouble understanding how to approach this problem, and would like to get some help and explanations :)
Let $X$ be a random variable with density $f_X(x)$. It is known that $X$ is non-...
- 189
2
votes
0
answers
25
views
Why does the distribution of entropy in a deck of shuffled cards not follow a normal distribution?
Around a year ago I did a study on shuffling cards and came across an unexpected result. I was investigating methods for quantifying how "shuffled" a deck of cards was, such as measuring the ...
- 520
0
votes
1
answer
50
views
Is $H(f(X))$ a concave function in $p(x)$?
$H(X)=-\sum_{x}p(x)\log {p(x)}$ is the entropy of a random variable $X$. I know that it is a concave function of $p(x)$. But is it still true that $H(f(X))$ is a concave function forall function $f(\...
- 164
1
vote
0
answers
63
views
Bernoulli Shift has complete positive entropy without using K-automorphism property
Let $X = (1,\dots,d)^{\mathbb{Z}}$ with the $\sigma$-álgebra $\beta $ generated by the cylinders and $\mu$ the product measure.
Let $T: X \rightarrow X$ be the left shifter, $T(x(n))=x(n+1)$.
We ...
0
votes
1
answer
75
views
Entropy reduction
May be this is an elementary question in information theory.
Given N random integers $x_1, x_2, \dots , x_N$ between 1 and m, what's the reduction in entropy if all integers are distinct?
1
vote
1
answer
48
views
Maximise entropy if we know the expected value of the distribution
Suppose $Z$ takes values in {0, 1, 2, · · · }. Given $E[Z] = a$ for some $a > 0$, find the probability
mass function $p_i = P(Z = i$) that maximises $H(Z)$
This reduces to an optimisation problem I ...
- 177
0
votes
0
answers
20
views
Effective computability of non-linear optimization algorithms
We are looking for any results on the effective computability of the optimization algorithms.
In particular, consider probability mass functions on a finite set $X=\{x_1, ... x_n\}$.
We are looking ...
- 1
1
vote
1
answer
61
views
How does the choice of alphabet impact the Shannon entropy of a sequence (if at all)?
MSA
The context for my problem is multiple sequence alignment, column entropy. So basically:
finding $H$ for sequences like MKR--KK-RR---RRM provided 1-letter code ...
- 215
2
votes
1
answer
45
views
The implication of $H(X_1,X_2,\dots,X_n) = nH(X_1)$
Let $X_1,X_2,\dots ,X_n$ be a discrete random process and $$a_n=\frac{1}{n}H(X_1,X_2,\dots ,X_n)$$Suppose further that we have for all $n\in \mathbb{N}$ $$a_n = a_{n+1}$$ I think this implies that $...
- 4,481
1
vote
2
answers
79
views
Find entropy of $p = \frac{1 + \delta}{2}$, where $|\delta| < 1$
I need to prove that for $p = \frac{1 + \delta}{2}$, where $|\delta| < 1$, entropy is gonna look like this
$$H(p) = \log2 \cdot \left(1 - \frac{1}{\log 4} \cdot \sum_{k\geq 1} \frac{\delta^{2k}}{k\...
