Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [entropy]

This tag is for questions about mathematical entropy. If you have a question about thermodynamical entropy, visit Physics Stack Exchange or Chemistry Stack Exchange instead.

0
votes
0answers
14 views

Finding the maximum entropy.

I'm trying to solve the following question: Here is my attempt using Lagrange multipliers: $L=-x_{1}lnx_{1}-x_{2}lnx_{2}-\cdots -x_{n}lnx_{n}+\lambda (x_{1}+\cdots +x_{n})$ $0=\frac{\partial L}{\...
3
votes
1answer
36 views

Entropy of the upper and lower bits of a square number

Consider a uniformly random number $x<2^n$. Let $H_{\star}(n)$ denote the (base-$2$ Shannon) entropy of the first $n$ bits of $x^2$, and let $H^{\star}(n)$ denote the entropy of the rest of the ...
0
votes
0answers
31 views

Metaphors for the entropy of a question

What would be appropriate metaphors to call the entropy of a question? I was thinking along the lines of "information value," but this would clearly be inappropriate, because it is the answers that ...
1
vote
0answers
14 views

Solution to maximum entropy of the gravity model

I need some algebra help to come up with a solution to the maximum entropy formulation of the gravity model. The problem: $ \hspace{35pt}\\ max \ H =-\sum_{i=1}^I \sum_{j=1}^J T{_i}_j\ln( T{_i}_j) \\ ...
0
votes
1answer
28 views

Optimization over Distributions

$$\min_{P, Q} E_{x \sim P} -\log \frac{Q(x)}{P(x) + Q(x)} + E_{x \sim Q} -\log \frac{P(x)}{P(x) + Q(x)}$$ For the above problem, what are the minimizer $P$, $Q$? Can we say that it is minimized only ...
1
vote
1answer
30 views

An inequality related to the Renyi divergence

Can you prove the following? Conjecture. Let $\lambda > 1$. Let $p_i$, $q_i$, $\mu_i$, $\nu_i$ be probability densities over $\mathbb R$ for $i = 1, ..., n$, such that for all $i = 1, ..., n$, (...
-1
votes
0answers
11 views

Entropy of Geometric Random variable

Let X be a geometrically distributed random variable, that is , 𝑃(𝑋 = 𝑘) = 𝑝(1 − 𝑝)^𝑘−1 , 𝑘 = 1,2,3 … 1) Find the entropy of X. 2) Knowing that 𝑋 > 𝐾, where K is a positive integer, what ...
2
votes
2answers
30 views

What is O in binomial distribution entropy?

I have searched everywhere but can't find an answer. What is "O" refers to in this equation? $${\frac {1}{2}}\log _{2}\left(2\pi enp(1-p)\right)+O\left({\frac {1}{n}}\right)$$
0
votes
0answers
21 views

Recursive Calculation of Entropy on A Process [closed]

A process X consists of first tossing a fair die, and if the result is 1, 2, 3 or 6, then the process stops. If the result is 4, then a biased coin with probability $\frac{1}{3}$ of ...
0
votes
1answer
26 views

Analysis of Entropy on Two Distributions: Proving $H(X) < H(X')$

Let $P=\{p_1, p_2, p_3 ..., p_n\}$ and $P^{'}= \left\{ \dfrac{(p_1 + p_2)}{2}, \dfrac{(p_1 + p_2)}{2}, p_3, ..., p_n\right\}$ be distributions on the same random variable $X$. $1$. Show $H(X)\leq H(X^...
0
votes
0answers
17 views

Calculating a discrete maximum entropy prior

Given a discrete random variable $\theta$ that takes values on $\{\theta_i\}_{1\leq i \leq m}$, with probability $\pi(\theta_i)$ the entropy is defined as $$ \mathcal{E}(\pi) = -\sum_i \pi(\theta_i)...
2
votes
1answer
107 views

Proof of convexity with logs

I'm trying to prove the following result. Let $p$ be an input probability distribution and $Q$ be a transition matrix. $q = Qp$ is a valid output probability distribution. The components of $p$ and $q$...
1
vote
2answers
37 views

How to compare the entropy of two systems?

Suppose I have two systems $A$ and $B$ that produces the numbered tiles. System $A$ produces tiles 1, 2, 3, 4, and 5 with the probabilities: ...
0
votes
1answer
56 views

Maximizing entropy

Let $v_1<\cdots<v_n$ and $\mu\in(v_1,v_n)$ be real numbers. Consider set $$X=\left\{(p_1,\ldots,p_n)\in[0,1]^n\ |\ \sum_{i=1}^np_i=1,\ \sum_{i=1}^np_iv_i=\mu\right\},$$ which is convex (easy) ...
0
votes
1answer
22 views

An inequality on mutual entropy

The question is as follows: Prove $H(X,Y,Z) - H(X,Y) \le H(X,Z)-H(X)$. Here, I tried to prove instead $$H(X,Z) - H(X) + H(X,Y)-H(X,Y,Z) \ge 0$$ I know that $H(X,Y,Z) = H(X,Y) + H(Z|X,Y)\ $ and $...
0
votes
0answers
24 views

Phase transitions in voting systems

Please, anyone could direct me a first-approach text in phase transitions of voting systems? I do not have any preliminary knowledge in statistical physics, Ising model and etc., I am only interested ...
0
votes
0answers
15 views

Proving a method to find the entropy of a measure-preserving function

I am reading chapter 10 this paper and it seems to state but not prove the following theorem. If P generates the σ-algebra of measurable sets, then $h_µ(T ) = h_µ(T,P )$. I understand everything in ...
0
votes
0answers
21 views

Entropy calculation between binary and geometric random variables

I can't solve an exercise. I don't know how to suggest it. Let X be a geometrically distributed random variable; i.e., $$\text{p(X=k) = $p(1-p)^{k-1}$ where k = 1,2,3,...}$$ with 0$\lt$p$\lt$1. a) ...
1
vote
1answer
20 views

For a Fixed Variance, Gaussian Distribution Maximizes Entropy?

I was reading this paper. In page 5, second column, they mention that, $$h(Q) + h(P) \ge log(e \pi) => \sigma(Q) * \sigma(P) >= \frac{1}{2}$$ Where entropy $h$ is defined in the following way:...
1
vote
3answers
46 views

Entropy of probability distributions

Let $\mu_1, \mu_2$ be two probability distributions on a sample space $X$ and let $0 < \alpha < 1$. Define the entropy of a probability distribution $\mu$ to be $$H(\mu) = - \sum_{t \in X} \mu(...
1
vote
0answers
31 views

Integral of Classical Entropy

I am reading some notes on Ricci flow by Peter Topping where the author introduces what he calls the 'classical entropy' for a function $u = e^{-f(t)}$: $$N= \int_M u \log(u)dV.$$ Using the fact ...
1
vote
0answers
31 views

Maximum entropy distribution given constrained minimum, maximum, and mean

What kind of distribution do we get if we constrain the range to be the unit interval and also constrain the mean to be $\alpha$? If we read this table, we see the following two examples of maximum ...
0
votes
1answer
23 views

Why do first and second moments insure stability?

In Pattern Recognition and Machine Learning, Bishop states "Let us now consider the maximum entropy configuration for a continuous variable. In order for this maximum to be well defined, it will ...
1
vote
1answer
40 views

Finding distribution to maximize entropy of a random variable subject to fixed mean

I saw this as a past paper question: $X$ is a random variable taking values in the positive integers, $\mathbb Z^{\geq 0}$, and has fixed mean $\mathbb E(X) = m$. Find the distribution of $X$ when ...
1
vote
0answers
20 views

Conditionnal entropy : intuitive interpretation

Consider two system $X$ and $Y$ described by probabilities distribution. We define the conditionnal entropy of $X$ knowing $Y$ as : $$S_{X|Y}=\sum_y p(y) \left( - \sum_{x} p(x|y) \log(p(x|y)) \right)...
0
votes
0answers
40 views

Finding the Information Gain in sets

I have a universe, $ U = \{a, b, c, d, e, f\}$ and sets $A = \{a, b, c\}$ and $B = \{a, d, e, f\}$ If $P(A) = P(X = x \in A)$ and $P(B) = P(X = x \in B)$, where $X$ is a random variable defined by ...
0
votes
1answer
24 views

Upper bound of Mutual Information

The Shannon entropy of a discrete random variable ${\textstyle X}$ with possible values ${\textstyle \left\{x_{1},\ldots ,x_{n}\right\}}$ and probability mass function ${\textstyle \mathrm {P} (X)}$ ...
0
votes
1answer
36 views

Log det of covariance and entropy

I understand log of determinant of covariance matrix bounds entropy for gaussian distributed data. Is this the case for non gaussian data as well and if so, why? What does Determinant of Covariance ...
0
votes
1answer
20 views

Entropy of a normal distribution in Bits versus Nats in book Elements of Information Theory

This should have been easy. Converting between nats and bits is a logarithmic change of base. So going from $\log$ base $e$ to base $2$, should require the denominator to have $\log_2(e)$. However in ...
0
votes
1answer
25 views

can mutual entropy be higher than joint entropy?

Let's assume I have three probability distributions A,B,C. The entropy of each is 1.58, with joint entropy 1.58. Calculating mutual entropy with formula I(A,B,C) = H(A)+H(B)+H(C)-J(A,B,C) results ...
2
votes
2answers
76 views

asymptotics for binomials

What is a published reference for the asymptotic equivalent of $ n \choose k$ with $k$ linear in $n$? I want both the entropy function and the denominator in $\sqrt{n}.$
0
votes
0answers
21 views

Binary Entropy Function (properties)

I'm not a mathematician and I'm trying to understand the properties of the binary entropy function. In particular, I would like to ask you, if the distributive law can be applied to the following ...
0
votes
0answers
22 views

Shannon Entropy of a periodic signal

We have a source $X$ with alphabet size equal to $N$. The Shannon entropy is defined as $$E(X)=-\sum _{i=1}^{N}p_{i}\cdot\log _{2}p_{i}$$ where $p_i$ is the probability of symbol $i$ appearing in the ...
1
vote
1answer
26 views

Mutual information between dependent and independent variables

This question is a follow-up for the previously asked question. Assume we have three random variables $A$, $B$, and $C$, where $A$ and $B$ are independent (i.e. $I(A \,; B)=0$), but the relation ...
5
votes
2answers
97 views

is Shannons “A mathematical theory of communication” worth reading for a beginner in information theory?

I'm studying information theory for the first time, chiefly through Cover&Thomas, in which entropy is introduced in the beginning of the first chapter, with the mathematical definition, but ...
1
vote
1answer
23 views

Mutual information with two independent variables

Let us say we have three random variables, $A$, $B$, and $C$, where $A$ and $B$ are independent. I know that $$I(A;B) = 0.$$ Also, my intuition is that $$I(A;B,C) = I(A;C).$$ However, I cannot either ...
2
votes
0answers
46 views

Oleinik condition is equivalent to Entropy Condition (PDE)

Show that for weak solutions of \begin{aligned} &u_t + f(u)_x = 0 \quad\text{in}\quad \mathbb{R}\times (0,\infty) \\ &u=u_0 \quad\text{on}\quad t=0 \end{aligned} The entropy condition ...
2
votes
2answers
48 views

Calculating the capacity of a noisy channel

Suppose we have a channel which transmits sequences of length n, of 0s and 1s (i.e. A={0,1} to the nth -> B={0,1} to the nth, such that during transmission, it will randomly (with equal probability) ...
-1
votes
1answer
27 views

Prove 0<=entropy<=1 [closed]

The entropy of a Bernoulli random variable X with $P(X=1)=q$ is given by B(q)=-qlog(q)-(1-q)log(1-q) How do we prove 0<=B(q)<=1?
0
votes
0answers
11 views

Effect of a convolution with a Bernoulli distribution on Rényi divergence

Let $P$ and $Q$ be two probability distributions on $\mathbb{Z}$. Let $D_\alpha(P\|Q)$ be the Rényi divergence of order $\alpha$ of $P$ and $Q$: $$ D_\alpha(P\|Q)=\frac{1}{\alpha-1}\sum_i\frac{P(i)^\...
3
votes
1answer
51 views

Trace of a matrix exponential with tensor products, and Von Neumann entropy

$\def\T{\operatorname{Tr}}$ $\def\1{\mathbb{1}}$ Let $H=H_1\otimes H_2\otimes H_3$ be a finite dimensional Hilbert space, and let $\rho_{123}$ be a self-adjoint matrix with $\rho_{123}\geq 0$ (...
0
votes
0answers
13 views

bounding min-entropy gain in differential privacy

In privacy-related computer science literature, we say that a randomized algorithm $\mathcal{K}$ that produces a model $\theta$ from a sample $X=(x_1,...,x_n)$ is $\epsilon$-differentially private iff ...
0
votes
0answers
19 views

Infimum of $f(\theta)= -\sum_{i=1}^n (y_i - \alpha p(y_i|x;\theta)) \ln p(y_i|x) $

Suppose $y_1 = 1, y_2 = y_3 = ...=y_n = 0\ (n\geq2)$, and $\sum_{i=1}^np(y_i|x;\theta) = 1$, $0\leq p(y_i|x;\theta)\leq1$. Meanwhile $\alpha > 0$ is a constant. Let's define a function $$f(\theta)...
1
vote
1answer
52 views

The motivation of continuous random variable mutual information by $k$'th nearest neighbour

I am reading this paper Kraskov et al, 2004, Estimating Mutual Information on estimating the mutual information of two continuous random variables based on entropy estimates from $k$-nearest neighbour ...
1
vote
1answer
50 views

Entropy for linear advection-diffusion equation

Let u be a solution to $$u_t+cu_x=\epsilon u_{xx}$$ with $\epsilon>0$, $c \in \mathbb{R}$. Show that for any convex function $\mu \in C^2(\mathbb{R})$ the total entropy $\int_\mathbb{R}\mu(u(x,t))...
0
votes
1answer
10 views

Can I derive a well-known distribution from another divergence?

A large class of distributions can be derived from $\max_{p(x)} H(p)$ s.t. $E_x{x}=\mu$ $E_x{x^n}=c_n$ where $H(p)$ denotes the Shannon (differential) entropy and are called maximum entropy. ...
0
votes
1answer
17 views

A hint for the entropy problem-entropy of one discrete variable is greater than the entropy of another one

I need a hint on how to start solving the following problem. Entropy of a discrete variable X is $H(X) = −\sum_{x\in \{x:P(X=x)>0\}}P(X=x)logP(X=x)$. Let $f:R → R$ be any function.\ a) Show that ...
0
votes
1answer
36 views

Distributing objects on containers

Suppose we have $n$ containers, each has the ability of holding $f_{i}$ object for $i=1, 2, \dots, n$. That means $f_{i}$ is the maximum number of object that the $i$th container can hold. Now, we ...
0
votes
0answers
28 views

take derivative of differential entropy of beta distribution

In order to find a beta prior for Bernoulli (Binomial) distribution, one way is to find the maximum entropy prior distribution. Now let assume if we only have information about the mean value of beta ...
0
votes
0answers
40 views

Bound on KL-divergence-like quantity (with squared logarithms)

Given two discrete probability distributions over $n$ events, with $p_i$ and $q_i$ denoting the probability that the ith event occurs respectively, I am looking for an upper bound of the following ...