Questions tagged [entropy]

This tag is for questions about mathematical entropy. If you have a question about thermodynamical entropy, visit Physics Stack Exchange or Chemistry Stack Exchange instead.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Non parametric estimation of point process entropy

Let point process $\Phi$ be defined on some bounded set $A$ in the borel $\sigma$-algebra on $\mathbb{R}^d$, and assume that the parametric structure of $\Phi$ is unknown. Let $\{x_1,\ldots,x_N\}$ be ...
1
vote
0answers
52 views

Normalizing constants preserve metric entropy

Suppose $\mathcal{F}=\left\{f\in L^2([a,b]): 0<\underline{c}\leq f\leq\overline{c} \right\}$. Consider the following transformation $$\tilde{\mathcal{F}} := \left\{\frac{f}{\int f d\mu}: f\in \...
2
votes
1answer
35 views

How to interpret that a distribution does not have defined entropy (or has infinite entropy)?

An entropy (is Shannon sense) can be interpreted as uncertainty or missing knowledge. When the knowledge is added, the entropy decreases. Hence it can also be interpreted as information content. ...
0
votes
0answers
21 views

Seeming abitrariness of the Maximum Entropy Distribution

I have a two parameter model $C = (C_1, C_2) \in \mathbb{R}^2$ and would like to look at the choice of parameters from a stochastic point of view. A minimal set of constraints on $C$ is that The mean ...
1
vote
1answer
52 views

Entropy of extractions

A box contains $3$ white and $6$ black balls. We draw $2$ balls consequentially without replacement. Find the entropy of first and second extractions and the entropy for both of them.
0
votes
0answers
12 views

What distribution used in practice does not have an entropy?

According to answer to my question Does any probability distribution have an entropy defined?, some distributions not having defined entropy can be constructed. Examples provided in the answer seems ...
0
votes
0answers
28 views

Inequality of factorials in the convex sum/multinomial form

Let $X,Y,Z$ be three random variables taking values in a finite set $S$ and satisfy $$\mathbb{P}(X=x,Y=y, Z=z)\cdot n\in \mathbb{N}$$ for all $x,y,z\in S$. Furthermore, we may assume that $n$ is much ...
3
votes
2answers
69 views

Show that $G\left(n,p\right)e^{-\frac{1}{12np\left(1-p\right)}}<{n\choose pn} < G\left(n, p\right)$

The problem: Using the inequalities $\sqrt{2\pi n}\left(\frac{n}{e}\right)^{n}e^{\left(\frac{1}{12n}-\frac{1}{360n^{3}}\right)} < n! < \sqrt{2\pi n}\left(\frac{n}{e}\right)^ne^{\frac{1}{12n}}$ ...
1
vote
2answers
50 views

Is the negative normalized entropy convex?

The negative normalized entropy is defined as $$h:\mathbb{R}_{>0}^n \rightarrow \mathbb{R} \ , \ h(x)=\sum_{i=1}^n x_i\log \frac{x_i}{\sum_{j=1}^n x_j} \ .$$ Is this function convex? Its Hessian is ...
0
votes
0answers
29 views

Derivation of exponential family of distributions solution to max entropy problem

In the derivation of exponential family distributions form as solutions to maximum entropy problem with linear constraints, the Lagrangian is computed as: $${\mathcal{L}}(p, \theta, \theta_0, \lambda) ...
0
votes
0answers
34 views

Entropy vs Relative Entropy.

Can anyone help me reconcile two well known observations. $\textbf{Observation 1}$ The Second Law of Thermodynamics says that a system wants to minimise (the negative of entropy) $$ H(\rho):=\int \...
0
votes
0answers
16 views

Conditional Entropy $H (X + z_1 | X + z_2)$ [closed]

How does the Conditional Entropy $H (X + z_1 | X + z_2)$ where $z_1$ and $z_2$ are independent RVs simplifies?
0
votes
1answer
17 views

Entropy with Markov Process

Can somebody please explain or help me reach an answer to this. I plainly don't know where to start. Sorry for not providing my work because am just confused on what to do. If somebody can please ...
2
votes
0answers
26 views

Optimal transport as a metric between two color images

I am trying to characterize a distance between two images in relation to the colors present in these images. Therefore I would like to solve Earth mover's distance/1-st Wasserstein distance with ...
1
vote
0answers
25 views

Why use KL-divergence in any practical setting?

The KL-divergence, see https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Definition, is defined as the expectation of the logarithmic difference of the probabilities of two ...
0
votes
0answers
22 views

Meaning of Relative Entropy

KL divergence means Expectation log of 1 likelihood by another ie., $E_{X \sim P_{\theta}}[- \log \frac{P_{\theta}(x)}{Q_{\phi}(x)}]$. Overall it means what is the average gap between the 2 ...
0
votes
0answers
11 views

Lower bound on entropy

Given a normalized state vector $\vec{v}(t)$ which varies continuously with time $t\geq0$ and a normalized probability density $P(t)$, where \begin{equation} \int_{0}^{\infty} P(t)dt = 1. \end{...
1
vote
2answers
61 views

Does any probability distribution have an entropy defined?

There are lot of probability distribution having infinite moments, for example Cauchy distribution has even the first moment infinite. Therefore, often we cannot calculate second moments used in ...
0
votes
1answer
26 views

Shannon entropy of languages

In his paper Prediction and Entropy of Printed English Shannon defines the entropy $H$ of a language as $$H = \lim_{N \to \infty} F_N$$ where $$F_N = \sum_{i, j} p(b_i, j) \log p(j | b_i)$$ where $b_i$...
1
vote
1answer
42 views

Sum of squared probability distribution $\sum_c p_c^2$ is a measure of uncertainty?

Can the sum of squared probabilities, i.e. $\sum_c p_c^2$, be considered to be a measure of uncertainty? If so, does it have a mathematical name or theory? The form is similar to Shannon's entropy, $-\...
0
votes
0answers
13 views

Stirling's approximation fractional error

Given Stirling's approximation $lnN!$ is approximated by $NlnN-N+\frac{1}{2}lnN$. I want to calculate the fractional error that comes from neglecting the third term $\frac{1}{2}lnN$ for N=10 and N=100....
0
votes
0answers
15 views

Information content of an inaccurate classification

I am trying to see how much information I am giving when I am making a classification. Say that there are $n$ classes, each with probability $p_i$. Now I have trained a classification model that ...
1
vote
1answer
33 views

Let $Z\sim N(0, 1)$ then $E(f(Z))\le E(f(X))$ for any convex $f$ and any $X$ with unit variance.

I thought about this question purely out of curiosity. I have a feeling that it is true, but I can't prove it. Standard normal distribution arises if we maximize the entropy among all absolutely ...
0
votes
1answer
73 views

Information lost in going from n-dice rolls to the sum of their faces?

I'm trying to understand how to figure out how much information is lost in going from n dice to the total score. If I rolled n-dice I could add up their face values to find the total sum, however if I ...
0
votes
0answers
18 views

What's the entropy of a dependent variable

It confuses me about the entropy of a dependent variable $y$ corresponding to the random variable $x$. Suppose $x$ is a variable with entropy as $H(x)$, and $y = f(x)$ is the dependent variable with ...
3
votes
3answers
139 views

Shuffle a poker deck between 4 players, with least required entropy

We are shuffling a standard poker deck of 52 cards between 4 players, each getting 13 cards. The order of cards for a particular player does not matter. A naive algorithm is to first shuffle the whole ...
0
votes
0answers
7 views

Lower bound for the convergence radius of the Poincaré serie

This is a technical follow up to this question of which I repeat the definition for clarity. Let $(X,g)$ be a Riemannian manifold on which a group $G$ acts properly by isometries $G\curvearrowright X$....
2
votes
0answers
22 views

Poincaré serie and partition functions

Let $(X,g)$ be a Riemannian manifold on which a group $G$ acts properly by isometries $G\curvearrowright X$. We can define the Poincaré serie for such action $$\mathcal{P}_s(G,X)=\sum_{g\in G^*}e^{-sd(...
0
votes
0answers
24 views

Show that a random variable is Shannon entropy

Consider a reference measure $\mathbb{P}_1$ and a squence of measures $\mathbb{P}_2(\theta)$ which contains a continuous parameter $\theta$. The RND \begin{align*} Z(\omega;\theta) = \...
2
votes
1answer
35 views

Mapping randomness to binary

I'm working on an algorithm, to map entropy from random events like coin flips, dice, user input, timing delays, and so on (any arbitrary random event), to binary, in a deterministic and reversible ...
0
votes
0answers
53 views

Doubts on some definition of Shannon's entropy notion

I would like some clarifications on two points of Shannon's definition of entropy for a random variable and his notion of self-information of a state of the random variable. We define the amount of ...
0
votes
0answers
24 views

Why does regularizing optimal transport with entropy improve the out-of-sample performance of learning algorithms?

\begin{equation} \mathcal{W}_\epsilon(\alpha, \beta) = \min_{\pi\in \Pi(\alpha\beta)} \int c(x,y) \mathrm{d}\pi(x,y) + \epsilon H(\pi(x,y)) \end{equation} is the entropy-regularized Wasserstein ...
0
votes
0answers
21 views

Does information entropy with binary options reduce to the simple frequency count?

This is a simple question, but I'd like some reassurance that my thinking is correct: I need to calculate Shannon information entropy and use it as a measure of randomness. I have a sequence of 200 ...
0
votes
0answers
46 views

How to compare the entropy of copulas?

Three different bivariate copulas are shown below with increasing degrees of dependence (parameter $\theta$). Differential entropy is a measure of disorder in a probability density like the copula. ...
1
vote
0answers
34 views

Expectation of a random cross entropy

Let $H(q,p)=-\sum_c q(c)\log p(c)$ the cross entropy between two different probability distributions on a finite set with cardinality $N$. $q$ and $p$ can be seen as elements of the $N-1$-symplex $S=\{...
2
votes
3answers
88 views

Solving Coin-Weighing Problem (81 Coins, 1 Fake) Using Information Theory

So I have a coin-weighing puzzle under these situations: There are 80 real coins and 1 fake coin (total of 81 coins). The real coins are all the same weight, and the weight of the fake coin is ...
3
votes
1answer
56 views

Measuring the Shannon Entropy of an ordered sequence

I have 927 unique sequences of the numbers 1, 2 and 3, all of which sum to 12 and represent every possible one-octave scale on the piano, with the numbers representing the intervals between notes in ...
7
votes
1answer
93 views

Entropy is subadditive

Let $(X,\mathcal{B},\mu)$ be a probability space, abd let $\alpha$ and $\beta$ be countable-measurable paratitions. Let $H_\mu(\alpha)$ and $H_\mu(\beta)$ be the (Shanon) entropy of $\alpha$ and $\...
1
vote
0answers
17 views

Is the average Shannon's entropy of two groups of variables the mean of their two respective entropy values?

I have two groups of variables that respectively have Shannon's entropy of value X and Y. Does it make sense to consider the average entropy of those two groups to be the mean between X and Y?
0
votes
0answers
25 views

How does negative copula entropy imply reduction in uncertainty like mutual-information?

Mutual information measures the reduction in uncertainty in a random variable $X$ from knowing additional information from another variable $Y$. If we instead measure the mutual information of $X$ ...
1
vote
1answer
40 views

How to compare two variables whose differential entropy are both negative?

I know that a higher positive value for entropy indicates greater uncertainty, but not sure how this works when comparing two negative values If two continuous random variables $X$ and $Y$ have ...
1
vote
1answer
29 views

How to comprehend positive measure entropy

How to comprehend the statements below? Let $X=[0,1],f_2(x)=2x(mod1)$.$\mu$ is a measure such that $f_2$ is ergodic.The positive entropy assumption $h_\mu(f_2)>0$ implies that the restriction of $...
2
votes
1answer
62 views

Estimating the Shanon entropy of a high-dimensional discrete multivariate random variable by dead simple monte carlo sampling?

I'm interested in estimating Shannon's entropy for a discrete multivariate random variable $X$ that has high dimensionality (i.e. $X=(X1,..,Xn)$, where n is at the hundreds). I can effectively sample ...
1
vote
1answer
41 views

How to show $3$-period point means positive entropy

Let $f$ be a continuous function from $ [0,1] $ to itself with a $3$-period point. How to show that the topology entropy of $f$ is positive?
4
votes
0answers
42 views

Entropy of the difference of two random variables

Let $X$ and $Y$ be two i.i.d (independent and identically distributed) discrete random variables with distribution $P=(p_0, p_1, \ldots, p_{q-1})$ and support $\{0,1,...,q-1\}$ with $q \geq 2$. Take $$...
3
votes
2answers
161 views

Symmetrizability of conservation laws via mathematical entropy

Let $$ \frac{\partial \boldsymbol{u}}{\partial t} + \sum_{j=1}^d \frac{\partial}{\partial x_j} \boldsymbol{f}_j(\boldsymbol{u}) = \boldsymbol{0}, $$ be a system of conservation laws and let us assume ...
1
vote
1answer
44 views

Monotone numerical scheme and entropy solution to conservation laws

A monotone numerical scheme is when I have two sequences $v=(v_j)_j$, $w=(w_j)_j$, and $v^{(n)} \ge w^{(n)}$ , then $v^{(n+1)} \ge w^{(n+1)}$ , where $ v_j^{(n)}$ is an appoximation of $u(x_j,t_n)$. ...
0
votes
0answers
95 views

Mutual Information for Continuous Random variables for system with 1 input and multiple outputs (SIMO System)

Let's consider channels of the form \begin{equation}\label{eq:channelmodel2} Y_{1}= X e^{j \theta_{1}}+ V_1 \end{equation} \begin{equation}\label{eq:channelmodel3} Y_{2}=X e^{j \theta_{2}}+ V_2 \end{...
1
vote
2answers
113 views

How to prove a differential entropy is not scale invariant?

For example, $S(X)=-E_X(\log(f_X))=-\int_{-\infty}^{+\infty}f_X(x)\log(f_X(x))dx$ A transformation of X changes the result:$S(aX)=S(X)+\log|a|$ and more in general $S(g(X))=S(X)+\int_{-\infty}^{+\...
0
votes
0answers
17 views

General (continuous + discrete) source coding theorem

I was wondering if someone could state and prove (or knew references that state and prove) Shannon's source coding theorem in a form that works both for continuous and discrete r.v.. It is very easy ...

1
2 3 4 5
25