# Questions tagged [entropy]

This tag is for questions about mathematical entropy. If you have a question about thermodynamical entropy, visit Physics Stack Exchange or Chemistry Stack Exchange instead.

1,652 questions
Filter by
Sorted by
Tagged with
23 views

### Approximation of entropy of binomial distribution

The approximation of entropy of binomial distribution is: $$\frac1 2 \log_2 \big( 2\pi e\, np(1-p) \big) + O \left( \frac{1}{n} \right)$$ Based on my understanding, this approximation is for large n ...
• 11
42 views

### Converge of iterated average posterior to high entropy distribution

Setup Assume $p_Y \in \Delta^n$ is a discrete probability distribution obtained by $p_Y=L_{Y|X}p_X$, where $L_{Y|X} \in \mathbb{R}^{n \times m}$ is an arbitrary likelihood (i.e, a column stochastic ...
26 views

### Shannon's finite state transducer (FST) entropy theorem

I am trying to make sense of the proof of Shannon's theorem that a finite state transducer cannot increase the entropy of its input. I would love some sort of drawing or intuitive formulation of it, ...
• 51
1 vote
38 views

### Doubts on "An Intensive Introduction to Cryptography" exercise about Shannon's entropy

I was going through the exercises in An Intensive Introduction to Cryptography (see full PDF here), and in particular, I had some doubts on Exercise 0.12 (found on page 42). Here is the relevant ...
• 158
96 views

### Approximating the Prime Counting Function as $\pi(x) \approx \frac{x^2}{\ln\left(\Gamma(x+1)\right)}$

Approximating the Prime Counting Function as $\boxed{\pi(x) \approx \frac{x^2}{\ln\left(\Gamma(x+1)\right)}}$ Intro________________ In a unrelated topic I was viewing how the mechanical statistics ...
• 1,634
8 views

• 85
18 views

### How can I derive equation 2.23(Newton-Raphson for Entropy) in NASA CEA analysis

It might sound a bit basic, but, I'd like to follow NASA CEA report I. analysis from the beginning. So, I have to derive the Newton-Raphson equation from the entropy equation, which is one of the ...
• 27
17 views

### KL Divergence larger than Conditional KL Divergence

Let $q(z|y)$ and $r(z)$ be variational approximations of $p(z|y)$ and $p(z)$, respectively. If I know that $H(q(z|y))=H(r(z))$, where $H$ is the entropy, I'd like to know if it is true that: \begin{...
• 13
56 views

### How to understand the calculations for the density matrices of a qubit state [closed]

From a lecture, I wrote this set of calculations for the entropy of a given qubit state. What I am unsure of, since it was not shown at the lecture, is how the lecturer came to each of these matrices. ...
• 2,973
30 views

### Diffusion PM: Remove the edge effect at t = 0

In DPM paper appendix B.2 for proving that the $\int d\mathrm{x}^{(0)} d\mathrm{x}^{(1)} q(\mathrm{x}^{(0)}, \mathrm{x}^{(1)}) log [\displaystyle \frac{\pi(\mathrm{x}^{(0)})}{\pi(\mathrm{x}^{(1)})}]$...
66 views

### Differentiating Entropy with respect to Convolution Parameters

First, some formula reminders for the sake of completion: $H(X) = -\sum_{i} p(x_i) \log p(x_i)$ is the entropy of a sequence $x_i$, where $p(x)$ is the discrete probability of x. A discrete ...
• 65
1 vote
36 views

• 11
1 vote
76 views

• 399
18 views

### Why we use Binarycrossentropy as loss function of logistic regression model?

Let's say we have a logistic regression model: $$z = \vec w \cdot \vec x + b$$ $$a_1 = g(z) = \frac{1}{1+e^{-z}} = P(y=1|\vec x)$$ $$a_2 = 1-a_1 = P(y=0|\vec x)$$ $$loss = -yln(a_1)-(1-y)ln(1-a_1)$$ ...
• 541
1 vote
33 views

• 2,115
162 views

• 4,852
32 views

### Twenty questions game where you have to guess two objects

Suppose we consider a variant of twenty questions game, where the answerer chooses two answers. On a query from the questioner, the answerer replies $0, 1$ or $2$, depending on the number of answers ...
• 2,081
41 views

### For a probability distribution is the difference between the largest and smallest probabilities bounded by the Shannon entropy?

Consider a set of probabilities $\{p_1, ..., p_N\}$ where $p_1\leq p_2\leq ... \leq p_N$. Now consider the difference between the largest and smallest probabilities $D =p_N-p_1$. I want to know if one ...
• 406
102 views

### Negative mutual information example. What's wrong about it?

I'm aware that by definition the Mutual Information (MI) should be non-negative, and there are two related questions here: (1) and (2). However, I can think of an example in Physics where it is (or at ...
• 355
51 views

### Maximum Entropy and Minimum Divergence

Let random variable $X$ be defined over alphabet $X = \{-2, 0, 2\}$. a) Find the distribution $p(x)$ that maximizes the entropy $H(X)$ while maintaining $E\{|X|\} = \theta$, where $\theta \in [0, 2]$. ...
• 21
61 views

### Find the copula to minimize KL-divergence [closed]

Consider the set of all N-dimensional copulas denoted by $\mathcal{C}^N$. Notice that each copula $C$ is a joint distribution on the state space $[0,1]^N$ with uniform marginal distributions. Now ...
• 11
1 vote
I have the following channel with $\mathcal X = \mathcal Y$: $p(y|x) = \begin{bmatrix} 1/2 & 1/2 & 0 \\ 0 & 1/2 & 1/2 \\ 1/2 & 0 & 1/2\end{bmatrix}$ Is it possible to ...