# Questions tagged [entropy]

This tag is for questions about mathematical entropy. If you have a question about thermodynamical entropy, visit Physics Stack Exchange or Chemistry Stack Exchange instead.

847 questions
14 views

28 views

### Optimization over Distributions

$$\min_{P, Q} E_{x \sim P} -\log \frac{Q(x)}{P(x) + Q(x)} + E_{x \sim Q} -\log \frac{P(x)}{P(x) + Q(x)}$$ For the above problem, what are the minimizer $P$, $Q$? Can we say that it is minimized only ...
30 views

### An inequality related to the Renyi divergence

Can you prove the following? Conjecture. Let $\lambda > 1$. Let $p_i$, $q_i$, $\mu_i$, $\nu_i$ be probability densities over $\mathbb R$ for $i = 1, ..., n$, such that for all $i = 1, ..., n$, (...
11 views

### Entropy of Geometric Random variable

Let X be a geometrically distributed random variable, that is , 𝑃(𝑋 = 𝑘) = 𝑝(1 − 𝑝)^𝑘−1 , 𝑘 = 1,2,3 … 1) Find the entropy of X. 2) Knowing that 𝑋 > 𝐾, where K is a positive integer, what ...
30 views

### What is O in binomial distribution entropy?

I have searched everywhere but can't find an answer. What is "O" refers to in this equation? $${\frac {1}{2}}\log _{2}\left(2\pi enp(1-p)\right)+O\left({\frac {1}{n}}\right)$$
21 views

### Recursive Calculation of Entropy on A Process [closed]

A process X consists of first tossing a fair die, and if the result is 1, 2, 3 or 6, then the process stops. If the result is 4, then a biased coin with probability $\frac{1}{3}$ of ...
26 views

10 views

### Can I derive a well-known distribution from another divergence?

A large class of distributions can be derived from $\max_{p(x)} H(p)$ s.t. $E_x{x}=\mu$ $E_x{x^n}=c_n$ where $H(p)$ denotes the Shannon (differential) entropy and are called maximum entropy. ...
17 views

### A hint for the entropy problem-entropy of one discrete variable is greater than the entropy of another one

I need a hint on how to start solving the following problem. Entropy of a discrete variable X is $H(X) = −\sum_{x\in \{x:P(X=x)>0\}}P(X=x)logP(X=x)$. Let $f:R → R$ be any function.\ a) Show that ...
36 views

### Distributing objects on containers

Suppose we have $n$ containers, each has the ability of holding $f_{i}$ object for $i=1, 2, \dots, n$. That means $f_{i}$ is the maximum number of object that the $i$th container can hold. Now, we ...
Given two discrete probability distributions over $n$ events, with $p_i$ and $q_i$ denoting the probability that the ith event occurs respectively, I am looking for an upper bound of the following ...