Tagged Questions

This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

1
vote
0answers
18 views

intuition for entropy solutions

For a hyperbolic PDE of the form $$u_t + f(u)_x = 0$$ it turns out that the right notion of solution is entropy solution. Now, the notion of classical solutions are obviously very natural, and also ...
0
votes
2answers
88 views

Entropy of sum of two Uniform random variables

say $X$ and $Y$ are two identical, independent and discrete Uniform random variables and $Z=X+Y$. I do not know more about the random variables. Assuming $H(\cdot)$ to be the entropy of a random ...
2
votes
1answer
41 views

Prove identity involving the Tsallis q-logarithm

The natural logarithm and the exponential can both be generalized to a called q-logarithms and q-exponentials.those functions are defined as follows: \begin{eqnarray} \log_q(x) &:=& ...
1
vote
0answers
39 views

What is the “true” entropy of a binary string?

Consider an infinite binary string $\sigma$ and define its entropy $$H_1 = -(p_0 \log_2 p_0 + p_1 \log_2 p_1)$$ with $p_i = \lim_{N\rightarrow \infty} N(i)/N$, $N(i)$ the number of $i$'s among the ...
0
votes
0answers
41 views

Problem with calculating probability of symbols

I've a $100 \times 100$ binary matrix it`s constructed with this probability table : i want to apply extended Huffman on this matrix my idea is to compress each column individually . - so starting ...
0
votes
0answers
16 views

Can we calculate entropy for nonstationary random variables?

Let's assume that $X$ is a discrete random variable, which can take any value from the set $\{x_0,\dots,x_n\}$ with the probability mass function $P(X)$. We can calculate the entropy of $X$ as ...
0
votes
0answers
28 views

Continuity of Shannon Entropy with respect to KL-Divergence distance

I am trying to prove the following statement which seems to be trivial but I cannot: Suppose $p$ and $q$ are two distributions. If $D(p||q) < \epsilon$ and $D(q||p) < \epsilon$ for some ...
1
vote
1answer
28 views

Generlized Entropy compared to Generalized Dimension

I am currently reading the following paper by F.Takens: Multifractal analysis of dimensions and entropies. This paper discusses two different measures. One is generalized entropies and the other is ...
-3
votes
1answer
81 views

What's the Shannon entropy of the prime numbers?

Here's a note that calculates it as 1. Do you know of any other calculations? http://www.math-math.com/2014/05/shannon-entropy-shannon-entropy-of.html
2
votes
1answer
44 views

necessity of uniform continuity for topological entropy

I am referring to Walters' book "Introduction to Ergodic Theory." When he defines the concept of topological entropy he always assumes that the transformation $T: X \rightarrow X$ is uniformly ...
0
votes
0answers
12 views

Comparing a vector with a sorted vector

I am currently looking at a database in which the tuples have a natural order in which each tuple has an integer ID that reflects its age. The more recent the tuple, the larger its ID. Now we are ...
3
votes
0answers
61 views

Huffman codes: does less entropy imply less weighted average codeword length?

Let $\Sigma$ be a source alphabet with a probability distribution over its symbols $P$. Then, the Shannon entropy of $\Sigma$ is $$-\sum p_j \times -\mbox{log}_2(p_j)$$ where $p_j$ is the probability ...
0
votes
1answer
42 views

Subadditivity of Entropy

We define $H(X) = -\sum_{x}p_{x}\log p_{x}$ and relative entropy as $H(p(x)||q(x)) = \sum_{x}p(x)\log \frac{p(x)}{q(x)} = -H(X)-\sum_{x}p(x)\log q(x).$ Now we have to prove that $H(X,Y,X) + H(Y) \leq ...
3
votes
2answers
56 views

Understanding conditional entropy intuitively $H[Y|X=x]$ vs $H[Y|X]$

I was trying to understand conditional entropy better. The part that was confusing me exactly was the difference between $H[Y|X=x]$ vs $H[Y|X]$. $E[Y|X=x]$ makes some sense to me intuitively because ...
0
votes
0answers
38 views

how can prove this statement?

According definition of Kullback–Leibler divergence, we have: $"k[f|g]=\int_{}^{} f(x)\log \frac{f(x)}{g(x)}\mathrm dx"$. Now How can I prove this statement:$$k[f|g]=k[f|w]k[w|g].$$ Thanks in advance. ...
2
votes
1answer
30 views

Are measures of information model specific?

Does an information measure for a signal do a better job if it assumes some things about the signal? For example: I have a digital stream of data, 0s and 1s coming at a clock rate $r$. What is the ...
1
vote
0answers
29 views

Solomonoff induction , Shannon Entropy, Kolmogorov Complexity.

If Expected Kolmogorov Complexity equals Shannon Entropy why can't Shannon Entropy be used as an approximation of Kolmogorov Complexity in Solomonoff Induction? Regarding Kolmogorov Complexity and ...
2
votes
0answers
39 views

toplogical entropy of general tent map

Measure theoretic entropy of General Tent maps The linked question made me wonder how to calculate the topological entropy of a general tent map. Let $I=[0,1]$ and $\alpha \in ( 0,1)$. Define $T: I ...
1
vote
2answers
87 views

Fano's inequality and error rate

The Wire-tap channel II (http://link.springer.com/chapter/10.1007%2F3-540-39757-4_5) article in proof of Theorem 1 uses Fano's inequality to estimate the entropy $H(S|\hat{S}) \leq K \cdot h(P_e)$ ...
5
votes
0answers
61 views

Looking for a a measure-theoretic treatment of “differential entropy”

If $X$ is a discrete random variable, its entropy $H(X)$ is usually defined as something along the lines of $-\sum \def\P{\mathbb{P}}\P(x) \log_2( \P(x))$, where the sum ranges over all the possible ...
1
vote
1answer
55 views

Shannon Entropy Minimization

The Shannon Entropy for an observation is given by $ -x \log_2(x)$. Why is the maximum entropy achieved at $x = \frac{1}{e}$, and not at $x = 0$? Could someone provide a logical explanation that ...
-1
votes
3answers
179 views

probabilistically what can we say about the next throw of a coin after n throws

this may sound easy or hard or whatever but i cant seem to find anything after searching around for a similar question/answer The question is this: What can we say (probabilistically) about the next ...
0
votes
1answer
42 views

For P0 close to P1 the relative entropy can be approximated by its series expansion,Why?

I am reading a article (An overview of distinguishing attacks on stream ciphers, Martin Hell · Thomas Johansson · Lennart Brynielsson) about Distinguishe Attacks. There is a approximate equation ...
2
votes
0answers
47 views

Maximize the expected Brier Score of P relative to P.

Fix a finite set $\mathcal S$ and let $\mathcal P$ be the collection of probability functions over the Boolean closure of $\mathcal S$. Let $\beta : \mathcal P \times \mathcal S \to \mathbb R$, with ...
5
votes
0answers
56 views

Decrease of entropy when iterating a random discrete function

Let $m$ be a positive integer. Let $S$ be the set of non-negative integers $x$ less than $m$, with $|S|=m$. Let $X_0$ be the discrete uniform distribution over $S$, with $P(x)=\begin{cases} 1/m & ...
0
votes
1answer
432 views

Correct algorithm for Shannon entropy with R

Shannon entropy is defined by: $H(X) = -\sum_{i} {P(x_i) \log_b P(x_i)}$, where b could be $e$, 2 or 10 (bit, nat, dit, respectively). My interpretation of the formula is: $H(X)$ is equal to the ...
1
vote
3answers
60 views

Does entropy $H(y)$ decrease as $H(x,y)$ decreases when $H(x)$ is fixed?

Can't find any proof in Shannon's 1948 paper. Can you provide one or disproof? Thank you. P.S. $H(x)$(or $H(y)$) is the entropy of messages produced by the discrete source $x$(or $y$). $H(x,y)$ is ...
0
votes
2answers
58 views

Wrong result from LLR using Dunning Entropy method

I'm trying to use Dunning's method of calculating LLR to compare word instances between two fulltext indexes. His method uses entropy as part of the calculation. Dunning's blog post: ...
3
votes
1answer
62 views

Topological Entropy of $(x,y)\mapsto(x+y,x+a)$

Let $a\in \mathbb T^1$, how can I calculate the topological entropies of the maps $T_1:(x,y)\mapsto(x+y,x+a)$ and $T_2 : (x,y) \mapsto (x+y,y+a)$ defined on $\mathbb T^2$. Here $\mathbb T^n$ is the ...
0
votes
2answers
34 views

Interpreting Password Entropy Calculation – Property of Character Entropy

I was reading this explanation on how to calculate the entropy of a password. The article is great and it explains it very succinctly that even I understood it. According to the site, if you have a ...
3
votes
0answers
63 views

Topological applications of topological entropy

I just learned topological entropy during a lecture about dynamical systems, and I wonder whether there exist purely topological applications of it.
2
votes
0answers
38 views

Joint entropy maximization with a constraint

So I have 2 random variables X and Y, where X can take on values {0,1,2,3} and Y can take on values {0,1,2,3,4}. I need to maximize H(X,Y) subject to the constraint that P(X≠Y)=0.5. This also gives ...
2
votes
3answers
528 views

Is Standard Deviation the same as Entropy?

We know that standard deviation (SD) represents the level of dispersion of a distribution. Thus a distribution with only one value (e.g., 1,1,1,1) has SD equals to zero. Similarly, such a distribution ...
1
vote
1answer
220 views

How is the entropy of the multivariate normal distribution with mean 0 calculated?

Here is what I have so far: $$\begin{align} h(x) &= - \int \frac{1}{(2\pi)^{\frac{D}{2}}\det\Sigma^{\frac{1}{2}}} \exp(-\frac{1}{2} x^T\Sigma^{-1}x) \ln ...
0
votes
3answers
80 views

Roll a dice, ignore result if result is maximum value

Let's say I have a six-faced dice, but I only want results between $1$ and $5$. One way to do that would be to roll a ten-faced dice and divide the result by two ($1-2$ becomes $1$, $3-4$ becomes ...
2
votes
0answers
53 views

Entropy of sum of random variables

Let $x_1,x_2,\dots,x_n$ by random variables which take the values $0$ or $1$ with $P(x_i = 1) = p_i$ and $P(x_i = 0) = 1-p_i$, where $0 \leq p_i \leq 1$ for $i=1,2,\dots, n$. Let $$X= \sum_{i=1}^n ...
1
vote
0answers
105 views

Continous wavelet transform and shannon Entropy.

Note: I have asked the same question on signal processing forum,but didn't get any answer. so it might be more like a math or physics question. Hope you don't consider it as cross-post. I am trying to ...
2
votes
0answers
25 views

probability subspaces that make entropy function equal to a constant value

Given the entropy fucntion: $$ H = -\sum_i^n p(i) \ln(p(i))\,.$$ where $p(i)$ are probabilities and $n=4$, I would like to know all the points in the probability space that make $H = k$, being $k$ a ...
0
votes
0answers
18 views

topological entropy of function with 3 period

How to prove that the topological entropy of the function with 3 period is strictly positive? I know that the function with 3 period implies chaos, but I cannot prove the topological entropy of it is ...
3
votes
3answers
162 views

Estimating the entropy

Given a discrete random variable $X$, I would like to estimate the entropy of $Y=f(X)$ by sampling. I can sample uniformly from $X$. The samples are just random vectors of length $n$ where the entries ...
1
vote
1answer
126 views

What is the maximum entropy distribution for a continuous random variable on $[0,\infty)$ with given mean and variance?

I know that for a given logmean and logstdev its the lognormal, but what about where we directly specify the mean and variance? The above seems to depend on the log-transformation to the maxent for ...
1
vote
1answer
32 views

Proving the monotonicity of a recurrence.

Define the following recurrence for $n = 1, 2, \cdots$ $T(n) = ( 1 - \operatorname{H}(\frac{1 - P^{\frac{1}{n}}}{2}))^n$ where $0 < P < 1$ is a constant, function $\operatorname{H}(\cdot)$ is ...
0
votes
2answers
48 views

Properties of joint entropy

I'm trying to show the following, but am stuck For discrete random variables $X$ and $Y$, show $$H(X,Y)\geq \max\{H(X),H(Y)\}$$ where $H$ represents entropy
1
vote
2answers
47 views

Entropy of a distribution over strings

Suppose for some parameter $d$, we choose a string from the Hamming cube ($\{0,1\}^d$) by setting each bit to be $0$ with probability $p$ and $1$ with probability $1-p$. What is the entropy of this ...
1
vote
1answer
58 views

Kullback–Leibler divergence in bits

Well known formula of KL divergence when we have a discrete probability distributions. $$D_{KL}(P \parallel Q)=\sum\limits_i \ln \left(\frac{P(i)}{Q(i)}\right) P(i)$$ Can someone explain why the ...
0
votes
0answers
55 views

Entropy rate of Gaussian stationary process

The entropy rate $H(X)$ of a stationary process $X_i$ satisfies $$H(X) = lim_{n\rightarrow \infty} H(X_n | X_{n-1},\dots,X_1)$$ If we have $X_i = AX_{i-1} + e_i$ where $e_i \sim N(0,1)$ and ...
0
votes
0answers
42 views

Entropy of a state in a Markov model

For a three-symbol source i need to calculate the entropy. The symbols in the source are a, b, c and the source can be modelled using a first order Markov model. I have the conditional probabilities ...
1
vote
1answer
54 views

Calculate entropy of modified base32 hash

I am trying to develop a scheme for generating unique (probable within bounds) ids in a distributed application. I want the id to be easily remembered, easily spoken, and easily read. I chose base32 ...
1
vote
0answers
74 views

what is relative entropy between to random binary string with length of $L_1$ & $L_2$?

I want calculate relative entropy between two strings of binary such as: $L_1:11000100011101001$ $L_2:00101110110111001$ It is primarily when the lengths of two strings is same and in general when ...
0
votes
1answer
27 views

$Y$ is a function of $X$: making an inference based on the markovity of $ X$

In the information theory book by Cover and Thomas it is written: if $X$ is markov and $Y$ is a function of $X$ then: ...