Skip to main content

Questions tagged [entropy]

This tag is for questions about mathematical entropy. If you have a question about thermodynamical entropy, visit Physics Stack Exchange or Chemistry Stack Exchange instead.

Filter by
Sorted by
Tagged with
0 votes
0 answers
23 views

Approximation of entropy of binomial distribution

The approximation of entropy of binomial distribution is: $$\frac1 2 \log_2 \big( 2\pi e\, np(1-p) \big) + O \left( \frac{1}{n} \right)$$ Based on my understanding, this approximation is for large n ...
MarcG's user avatar
  • 11
0 votes
0 answers
42 views

Converge of iterated average posterior to high entropy distribution

Setup Assume $p_Y \in \Delta^n$ is a discrete probability distribution obtained by $p_Y=L_{Y|X}p_X$, where $L_{Y|X} \in \mathbb{R}^{n \times m}$ is an arbitrary likelihood (i.e, a column stochastic ...
backboltz37's user avatar
0 votes
0 answers
26 views

Shannon's finite state transducer (FST) entropy theorem

I am trying to make sense of the proof of Shannon's theorem that a finite state transducer cannot increase the entropy of its input. I would love some sort of drawing or intuitive formulation of it, ...
iuerlhgw's user avatar
1 vote
1 answer
38 views

Doubts on "An Intensive Introduction to Cryptography" exercise about Shannon's entropy

I was going through the exercises in An Intensive Introduction to Cryptography (see full PDF here), and in particular, I had some doubts on Exercise 0.12 (found on page 42). Here is the relevant ...
chirpyboat73's user avatar
2 votes
1 answer
96 views

Approximating the Prime Counting Function as $\pi(x) \approx \frac{x^2}{\ln\left(\Gamma(x+1)\right)}$

Approximating the Prime Counting Function as $\boxed{\pi(x) \approx \frac{x^2}{\ln\left(\Gamma(x+1)\right)}}$ Intro________________ In a unrelated topic I was viewing how the mechanical statistics ...
Joako's user avatar
  • 1,634
0 votes
0 answers
8 views

Is a concave parametric curve along a concave surface guaranteed to be concave along another concave surface?

Take a parametrized probability distribution $\mathbf{p}(\theta)=( p_0(\theta),p_1(\theta),\cdots p_n(\theta))$ and two permutation-symmetric, everywhere-concave functions $S_1(\mathbf{p})$ and $S_2(\...
Quantum Mechanic's user avatar
1 vote
2 answers
77 views

Von Neumann entropy vs Shannon entropy

Let us consider a mixture of quantum states $$ \rho = \sum p_{i}\left\vert \psi_i\right\rangle \left\langle\psi_i\right\vert\quad \mbox{probability distribution}\,\,\, p_{i} $$ If the $\psi_{i}$ form ...
Jip's user avatar
  • 21
0 votes
1 answer
58 views

Is the entropy of a distribution that follows the exponential family differential equation always concave in the natural parameter?

Are there any proofs, conjectures, counterexamples, or other helpful references related to the titular question? To clarify, let the entropy of a random variable $X$ distributed according to a ...
nlupugla's user avatar
4 votes
2 answers
293 views

Prove that $p = 1/2$ maximizes the entropy of a binomial distribution. [closed]

I am trying to find a proof that the entropy of a Binomial distribution is maximized at $p = 1/2$. By symmetry, I can easily show that $p = 1/2$ is a local optimum, but I'm stuck trying to show that $...
nlupugla's user avatar
0 votes
0 answers
18 views

How can I derive equation 2.23(Newton-Raphson for Entropy) in NASA CEA analysis

It might sound a bit basic, but, I'd like to follow NASA CEA report I. analysis from the beginning. So, I have to derive the Newton-Raphson equation from the entropy equation, which is one of the ...
dave's user avatar
  • 27
0 votes
0 answers
17 views

KL Divergence larger than Conditional KL Divergence

Let $q(z|y)$ and $r(z)$ be variational approximations of $p(z|y)$ and $p(z)$, respectively. If I know that $H(q(z|y))=H(r(z))$, where $H$ is the entropy, I'd like to know if it is true that: \begin{...
AAA's user avatar
  • 13
-2 votes
1 answer
56 views

How to understand the calculations for the density matrices of a qubit state [closed]

From a lecture, I wrote this set of calculations for the entropy of a given qubit state. What I am unsure of, since it was not shown at the lecture, is how the lecturer came to each of these matrices. ...
Superunknown's user avatar
  • 2,973
0 votes
0 answers
30 views

Diffusion PM: Remove the edge effect at t = 0

In DPM paper appendix B.2 for proving that the $\int d\mathrm{x}^{(0)} d\mathrm{x}^{(1)} q(\mathrm{x}^{(0)}, \mathrm{x}^{(1)}) log [\displaystyle \frac{\pi(\mathrm{x}^{(0)})}{\pi(\mathrm{x}^{(1)})}] $...
Zohreh Adabi's user avatar
0 votes
2 answers
66 views

Differentiating Entropy with respect to Convolution Parameters

First, some formula reminders for the sake of completion: $H(X) = -\sum_{i} p(x_i) \log p(x_i)$ is the entropy of a sequence $x_i$, where $p(x)$ is the discrete probability of x. A discrete ...
2 False's user avatar
  • 65
1 vote
2 answers
36 views

Minimizing KL-divergence against un-normalized probability distribution

Let $\Delta_n$ denote the $n$-dimensional simplex, ie. $\Delta_n \equiv \left\{ x \in \mathbb{R}^n_+ : \sum_{i=1}^n x_i = 1 \right\}$. I have a strictly positive (component-wise) vector $w \in \mathbb{...
msantama's user avatar
  • 1,213
1 vote
0 answers
29 views

Understanding of KL divergence

I am learning machine learning and encountered KL divergence: $$ \int p(x) \log\left(\frac{p(x)}{q(x)}\right) \, \text{d}x $$ I understand that this measure calculates the difference between two ...
Dmitry_IT_03's user avatar
1 vote
1 answer
98 views

Seeking Advice on Teaching Shannon Entropy to High School Seniors

I am planning to give a lesson on Shannon entropy to high school seniors who are likely to pursue scientific studies at university next year. As part of the lesson, I want to provide a practical ...
Mark's user avatar
  • 7,880
0 votes
0 answers
29 views

How to find "extra" mutual information

Codewords $00$ and $11$ are sent with equal probability through a BSC with error probability p. Compute the mutual information between the codeword sent and the first digit received as output. I have ...
S. Chitratta's user avatar
1 vote
1 answer
59 views

Differential Entropy of the Dirac Delta Function

We are working on finding differential entropy of Dirac delta function $\delta(x)$, that is $$H(X)= - \int_{-\infty}^{\infty} \delta(x) \ln \delta(x) \, \mathrm{d}x$$ We found answer that $H(X)= -\...
Jay's user avatar
  • 11
1 vote
1 answer
76 views

$H(X|Y)=H(X)\Rightarrow X\perp Y $

I need to prove that $ H\left(X|Y\right)=H\left(X\right)\Rightarrow X\perp Y $ where $X\perp Y $ means they are independent. $$ H\left(X|Y\right)=-\sum_{x}\sum_{y}P\left(x,y\right)\log_{2}P\left(x|y\...
Danny Blozrov's user avatar
0 votes
0 answers
12 views

How to calculate the upper bound of the difference between Renyi entropies of different orders?

The definition of Renyi entropy is: $$H_\alpha(X)=\frac{1}{1-\alpha}log\sum_{i}p_i^\alpha$$. My question is, given different $\alpha$, for example, $\alpha_1=1$ and $\alpha_2=2$, what is the upper ...
Zhongxia Shang's user avatar
2 votes
0 answers
71 views

Mutual information when sampling a random variable multiple times

Let $X$ be a random variable. For a fixed (known) preparation of $X$, suppose I have a protocol that generates a second random variable, $Y$, in a way that indirectly depends on $X$. Ultimately the ...
smitke6's user avatar
  • 699
0 votes
1 answer
38 views

Estimating the conditional entropy of a discrete variable conditioning on continuous variable

I am doing a machine learning project and I am trying to select the best features by computing their mutual information and select the ones with the highest information gain. I was looking at this ...
Ishigami's user avatar
  • 1,727
1 vote
1 answer
41 views

Symmetric difference of Shannon's entropy satisfies triangle inequality

For random variables $X$ and $Y$, we define: $$\delta(X,Y) = H(X|Y) + H(Y|X)$$ Show that $\delta(X,Y)$ satisfies the triangle inequality. My attempt: I tried to write $H(X|Y) = H(X,Y) - H(Y)$, then I ...
vegetandy's user avatar
  • 305
0 votes
1 answer
39 views

Does Entropy really change depending on the encoding? [closed]

So I'm self studying information theory, and I have a few doubts on entropy and encoding as a whole. I'm trying to compress a simple 16bit signed int sequence of values the best I can. I learned about ...
2 False's user avatar
  • 65
0 votes
1 answer
15 views

About the derivative of the conditional Rényi entropy

Preliminary considerations: In the paper unifying framework of information measures the conditional exponential entropy (see equation 29) is defined as: \begin{equation} \mathcal{E}_{\alpha}(X|Y) = ...
Upax's user avatar
  • 2,115
0 votes
0 answers
10 views

What is a flat measure and and how does it change the entropy of a Gaussian?

I'm reading page 6 of this paper written by David MacKay, and equation 3.2 has this funny $m^2$ which I don't understand. Essentially, we use a second-order approximation $M^*(w)$ of an objective ...
Snowball's user avatar
  • 1,119
1 vote
0 answers
21 views

Explanation of the notation for probability measures in the context of the Kullback-Leibler divergence sought

I came across some notation in the Wikipedia article https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence on the Kullback-Leibler divergence that I am unfamiliar with, in which the ...
Martin Vaughan's user avatar
0 votes
0 answers
13 views

Conditional von neumann entropy

Can someone show me how to proof this relation: If $\rho_{ABC}$ is a pure state, then $S(A|C)=-S(A|B)$ I already know that $S(A|B)=S(\rho_{AB})-S(\rho_{B})$ and that I should somehow use that $S(\rho_{...
Synonym's user avatar
3 votes
3 answers
491 views

Why are probabilities weighed by their logarithms in the definition of entropy? How does that relate entropy to “surprise”?

I am trying to understand the logic behind the mathematical formula for Entropy: $$ \text{ Entropy for a Discrete Random Variable:} \quad H(X) = -\sum_{x \in X} p(x) \log_2 p(x) $$ $$ \text{ Entropy ...
wulasa's user avatar
  • 399
0 votes
0 answers
18 views

Why we use Binarycrossentropy as loss function of logistic regression model?

Let's say we have a logistic regression model: $$z = \vec w \cdot \vec x + b$$ $$a_1 = g(z) = \frac{1}{1+e^{-z}} = P(y=1|\vec x)$$ $$a_2 = 1-a_1 = P(y=0|\vec x)$$ $$loss = -yln(a_1)-(1-y)ln(1-a_1) $$ ...
samsamradas's user avatar
1 vote
1 answer
33 views

Optimizing entropy of conditional Gaussians

this is a very specific problem, but i don't really know who to turn to to ask if my logic is correct, hence the post. My problem is the following: Let $V\in \{1,2\}$ be a random variable such that $P(...
Victor's user avatar
  • 11
1 vote
0 answers
232 views

About the chain rule of the exponential entropy

In the paper unifying framework of information measures the conditional exponential entropy (see equation 29) is defined as: $\mathcal{E}_{\alpha}(X|Y) = E_y\left(\int_{\mathbb{R}} f^{\alpha}(x|y)\,...
Upax's user avatar
  • 2,115
0 votes
0 answers
162 views

About the monotonicity of the exponential entropy

In the paper The Unifying Frameworks of Information Measures the conditional exponential entropy (see equation 29) is defined as: $\mathcal{E}_{\alpha}(X|Y) = E_y\left(\int_{\mathbb{R}} f^{\alpha}(x|y)...
Upax's user avatar
  • 2,115
2 votes
3 answers
83 views

Shannon source coding theorem and differential entropy

Loosely speaking, Shannon's source encoding theorem says that there is an encoder with rate at least $H(x)$ such that $n$ repetitions of the source can be mapped to at least $nH(x)$ bits, such that ...
nervxxx's user avatar
  • 281
0 votes
1 answer
41 views

$H(X) = H(Y) \stackrel{?}{\implies} H(f(X)) = H(f(Y))$ for a determinsitic function $f$

I want to prove that if the entropy of two random variables, defined on the same space, is the same, then the entropy of any deterministic function of the respective random variables will also have ...
QED's user avatar
  • 12.7k
1 vote
1 answer
29 views

Relative entropy and finite second moment bound

Suppose that $\nu$ is a probability measure (say on $\mathbb{R}$) with finite second moment. Let $\mu$ be another probability measure. Suppose that $KL(\mu \mid \nu)<\infty$. Does this then imply ...
pseudocydonia's user avatar
0 votes
0 answers
39 views

entropy of weighted sampling without replacement

Let us have $k$ elements of weights $w_1,…,w_k$. We sample elements with weights without replacement. At the first, the probability of element $i$ to be sampled is $w_i / \sum_{l=1}^k w_l$. Then, the ...
Wonbin Kweon's user avatar
1 vote
2 answers
46 views

Why does block coding via typical strings give messages longer than $nH(p)$?

This semester, I am taking a course on quantum information and quantum computing. Since I am rather new to information theory I have a problem with understanding a paragraph in my lecture notes. The ...
luki luk's user avatar
  • 189
0 votes
1 answer
63 views

Can you use entropy to find good splits in Guess Who?

In the game Guess Who, there are 24 characters with various traits. The objective is to guess the character of your opponent. I was wondering if it makes sense to use entropy as a measure of ...
hhh3's user avatar
  • 53
2 votes
1 answer
74 views

Topological entropy of a Bernoulli Shift

I am approaching the world of entropy and I would like to have a few examples in mind. I know some chaotic systems that are topologically conjugated to the Bernoulli shift, so I would like to know the ...
Andrea Marino's user avatar
1 vote
1 answer
42 views

Showing that the Shannon entropy is a special case of the von Neumann entropy

I would like to show that the Shannon entropy ($\mathcal{S}(\rho)=-\sum_j p_j\ln p_j$) is a special case of the von Neumann entropy, $\mathcal{D}(\rho)=-\text{Tr}\rho \ln\rho$. i.e. $$-\text{Tr}\rho \...
Superunknown's user avatar
  • 2,973
-1 votes
1 answer
54 views

Supposed to be a simple question about entropy

Let say there is an urn which contains balls of different color. It is a well known formula to calculate entropy of balls in the urn: H = - sum Pi*log(Pi) where Pi = Mi/N, where Mi - number of balls ...
irriss's user avatar
  • 99
0 votes
0 answers
29 views

Discrete analogue of entropy-variance inequality

It is well-known that the normal distribution maximizes entropy among all probability distributions on $\mathbb{R}^n$ with fixed mean and covariance. Since the entropy of the normal distribution $\...
Levent's user avatar
  • 4,852
3 votes
1 answer
32 views

Twenty questions game where you have to guess two objects

Suppose we consider a variant of twenty questions game, where the answerer chooses two answers. On a query from the questioner, the answerer replies $0, 1$ or $2$, depending on the number of answers ...
zxcvber's user avatar
  • 2,081
2 votes
0 answers
41 views

For a probability distribution is the difference between the largest and smallest probabilities bounded by the Shannon entropy?

Consider a set of probabilities $\{p_1, ..., p_N\}$ where $p_1\leq p_2\leq ... \leq p_N$. Now consider the difference between the largest and smallest probabilities $D =p_N-p_1$. I want to know if one ...
asph's user avatar
  • 406
0 votes
1 answer
102 views

Negative mutual information example. What's wrong about it?

I'm aware that by definition the Mutual Information (MI) should be non-negative, and there are two related questions here: (1) and (2). However, I can think of an example in Physics where it is (or at ...
Girardi's user avatar
  • 355
2 votes
1 answer
51 views

Maximum Entropy and Minimum Divergence

Let random variable $X$ be defined over alphabet $X = \{-2, 0, 2\}$. a) Find the distribution $p(x)$ that maximizes the entropy $H(X)$ while maintaining $E\{|X|\} = \theta$, where $\theta \in [0, 2]$. ...
learner's user avatar
  • 21
0 votes
1 answer
61 views

Find the copula to minimize KL-divergence [closed]

Consider the set of all N-dimensional copulas denoted by $\mathcal{C}^N$. Notice that each copula $C$ is a joint distribution on the state space $[0,1]^N$ with uniform marginal distributions. Now ...
copula's user avatar
  • 11
1 vote
1 answer
42 views

Is it possible to decrease the channel capacity by adding a row to the coding channel matrix?

I have the following channel with $\mathcal X = \mathcal Y$: $ p(y|x) = \begin{bmatrix} 1/2 & 1/2 & 0 \\ 0 & 1/2 & 1/2 \\ 1/2 & 0 & 1/2\end{bmatrix} $ Is it possible to ...
Danny's user avatar
  • 51

1
2 3 4 5
34