This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

1
vote
1answer
53 views

Shannon Entropy Minimization

The Shannon Entropy for an observation is given by $ -x \log_2(x)$. Why is the maximum entropy achieved at $x = \frac{1}{e}$, and not at $x = 0$? Could someone provide a logical explanation that ...
-1
votes
3answers
156 views

probabilistically what can we say about the next throw of a coin after n throws

this may sound easy or hard or whatever but i cant seem to find anything after searching around for a similar question/answer The question is this: What can we say (probabilistically) about the next ...
0
votes
1answer
39 views

For P0 close to P1 the relative entropy can be approximated by its series expansion,Why?

I am reading a article (An overview of distinguishing attacks on stream ciphers, Martin Hell · Thomas Johansson · Lennart Brynielsson) about Distinguishe Attacks. There is a approximate equation ...
2
votes
0answers
45 views

Maximize the expected Brier Score of P relative to P.

Fix a finite set $\mathcal S$ and let $\mathcal P$ be the collection of probability functions over the Boolean closure of $\mathcal S$. Let $\beta : \mathcal P \times \mathcal S \to \mathbb R$, with ...
5
votes
0answers
52 views

Decrease of entropy when iterating a random discrete function

Let $m$ be a positive integer. Let $S$ be the set of non-negative integers $x$ less than $m$, with $|S|=m$. Let $X_0$ be the discrete uniform distribution over $S$, with $P(x)=\begin{cases} 1/m & ...
0
votes
1answer
219 views

Correct algorithm for Shannon entropy with R

Shannon entropy is defined by: $H(X) = -\sum_{i} {P(x_i) \log_b P(x_i)}$, where b could be $e$, 2 or 10 (bit, nat, dit, respectively). My interpretation of the formula is: $H(X)$ is equal to the ...
1
vote
3answers
55 views

Does entropy $H(y)$ decrease as $H(x,y)$ decreases when $H(x)$ is fixed?

Can't find any proof in Shannon's 1948 paper. Can you provide one or disproof? Thank you. P.S. $H(x)$(or $H(y)$) is the entropy of messages produced by the discrete source $x$(or $y$). $H(x,y)$ is ...
0
votes
1answer
44 views

Wrong result from LLR using Dunning Entropy method

I'm trying to use Dunning's method of calculating LLR to compare word instances between two fulltext indexes. His method uses entropy as part of the calculation. Dunning's blog post: ...
3
votes
1answer
60 views

Topological Entropy of $(x,y)\mapsto(x+y,x+a)$

Let $a\in \mathbb T^1$, how can I calculate the topological entropies of the maps $T_1:(x,y)\mapsto(x+y,x+a)$ and $T_2 : (x,y) \mapsto (x+y,y+a)$ defined on $\mathbb T^2$. Here $\mathbb T^n$ is the ...
0
votes
2answers
32 views

Interpreting Password Entropy Calculation – Property of Character Entropy

I was reading this explanation on how to calculate the entropy of a password. The article is great and it explains it very succinctly that even I understood it. According to the site, if you have a ...
2
votes
0answers
54 views

Topological applications of topological entropy

I just learned topological entropy during a lecture about dynamical systems, and I wonder whether there exist purely topological applications of it.
2
votes
0answers
32 views

Joint entropy maximization with a constraint

So I have 2 random variables X and Y, where X can take on values {0,1,2,3} and Y can take on values {0,1,2,3,4}. I need to maximize H(X,Y) subject to the constraint that P(X≠Y)=0.5. This also gives ...
2
votes
3answers
251 views

Is Standard Deviation the same as Entropy?

We know that standard deviation (SD) represents the level of dispersion of a distribution. Thus a distribution with only one value (e.g., 1,1,1,1) has SD equals to zero. Similarly, such a distribution ...
1
vote
1answer
113 views

How is the entropy of the multivariate normal distribution with mean 0 calculated?

Here is what I have so far: $$\begin{align} h(x) &= - \int \frac{1}{(2\pi)^{\frac{D}{2}}\det\Sigma^{\frac{1}{2}}} \exp(-\frac{1}{2} x^T\Sigma^{-1}x) \ln ...
0
votes
3answers
77 views

Roll a dice, ignore result if result is maximum value

Let's say I have a six-faced dice, but I only want results between $1$ and $5$. One way to do that would be to roll a ten-faced dice and divide the result by two ($1-2$ becomes $1$, $3-4$ becomes ...
2
votes
0answers
52 views

Entropy of sum of random variables

Let $x_1,x_2,\dots,x_n$ by random variables which take the values $0$ or $1$ with $P(x_i = 1) = p_i$ and $P(x_i = 0) = 1-p_i$, where $0 \leq p_i \leq 1$ for $i=1,2,\dots, n$. Let $$X= \sum_{i=1}^n ...
1
vote
0answers
93 views

Continous wavelet transform and shannon Entropy.

Note: I have asked the same question on signal processing forum,but didn't get any answer. so it might be more like a math or physics question. Hope you don't consider it as cross-post. I am trying to ...
2
votes
0answers
24 views

probability subspaces that make entropy function equal to a constant value

Given the entropy fucntion: $$ H = -\sum_i^n p(i) \ln(p(i))\,.$$ where $p(i)$ are probabilities and $n=4$, I would like to know all the points in the probability space that make $H = k$, being $k$ a ...
0
votes
0answers
17 views

topological entropy of function with 3 period

How to prove that the topological entropy of the function with 3 period is strictly positive? I know that the function with 3 period implies chaos, but I cannot prove the topological entropy of it is ...
3
votes
2answers
134 views

Estimating the entropy

Given a discrete random variable $X$, I would like to estimate the entropy of $Y=f(X)$ by sampling. I can sample uniformly from $X$. The samples are just random vectors of length $n$ where the entries ...
1
vote
1answer
100 views

What is the maximum entropy distribution for a continuous random variable on $[0,\infty)$ with given mean and variance?

I know that for a given logmean and logstdev its the lognormal, but what about where we directly specify the mean and variance? The above seems to depend on the log-transformation to the maxent for ...
1
vote
1answer
28 views

Proving the monotonicity of a recurrence.

Define the following recurrence for $n = 1, 2, \cdots$ $T(n) = ( 1 - \operatorname{H}(\frac{1 - P^{\frac{1}{n}}}{2}))^n$ where $0 < P < 1$ is a constant, function $\operatorname{H}(\cdot)$ is ...
0
votes
2answers
46 views

Properties of joint entropy

I'm trying to show the following, but am stuck For discrete random variables $X$ and $Y$, show $$H(X,Y)\geq \max\{H(X),H(Y)\}$$ where $H$ represents entropy
1
vote
2answers
44 views

Entropy of a distribution over strings

Suppose for some parameter $d$, we choose a string from the Hamming cube ($\{0,1\}^d$) by setting each bit to be $0$ with probability $p$ and $1$ with probability $1-p$. What is the entropy of this ...
1
vote
1answer
53 views

Kullback–Leibler divergence in bits

Well known formula of KL divergence when we have a discrete probability distributions. $$D_{KL}(P \parallel Q)=\sum\limits_i \ln \left(\frac{P(i)}{Q(i)}\right) P(i)$$ Can someone explain why the ...
0
votes
0answers
44 views

Entropy rate of Gaussian stationary process

The entropy rate $H(X)$ of a stationary process $X_i$ satisfies $$H(X) = lim_{n\rightarrow \infty} H(X_n | X_{n-1},\dots,X_1)$$ If we have $X_i = AX_{i-1} + e_i$ where $e_i \sim N(0,1)$ and ...
0
votes
0answers
41 views

Entropy of a state in a Markov model

For a three-symbol source i need to calculate the entropy. The symbols in the source are a, b, c and the source can be modelled using a first order Markov model. I have the conditional probabilities ...
1
vote
1answer
51 views

Calculate entropy of modified base32 hash

I am trying to develop a scheme for generating unique (probable within bounds) ids in a distributed application. I want the id to be easily remembered, easily spoken, and easily read. I chose base32 ...
1
vote
0answers
68 views

what is relative entropy between to random binary string with length of $L_1$ & $L_2$?

I want calculate relative entropy between two strings of binary such as: $L_1:11000100011101001$ $L_2:00101110110111001$ It is primarily when the lengths of two strings is same and in general when ...
0
votes
1answer
26 views

$Y$ is a function of $X$: making an inference based on the markovity of $ X$

In the information theory book by Cover and Thomas it is written: if $X$ is markov and $Y$ is a function of $X$ then: ...
0
votes
2answers
49 views

Can infinite random sequences be asymptotically compressed?

A number $0.5<p<1$ is chosen at random and given to two people A and B whom are allowed to communicate before beeing separated. A is then given a sequence S of N random bits where each bit has ...
1
vote
1answer
68 views

Maximising Entropy of Random Variable taking Positive Integer Values

A random variable $X$ takes positive integer values and $E[X]=6$. What distribution of the random variable $X$ maximises the entropy $H(X)$? What if $X$ can only take a finite number of values? ...
3
votes
1answer
115 views

What does SVD Entropy Capture?

Looking at different definitions and types of Entropy, I run into the concept of SVD Entropy, which is defined as explained below. What is the intuition behind the SVD spectrum? What do different ...
2
votes
1answer
49 views

How to prove the function is logarithmic with coefficience

I am given a set of properties for an unknown function $f(x)$. In particular, not constantly zero, not negative, additive and continues for any $x$ between 0 and 1. I am asked to show equivalence ...
2
votes
1answer
74 views

Why $p$ factor in $p \log p$ in the entropy formula?

It is explained that we have log for additivity of information in the entropy formula. But, why is the $p$ factor? It is redundant, since we already have it in the $\log p$!
2
votes
0answers
128 views

How many code words if average code length equals entropy

I've been given a proof of the following: If $q\geq2$, then there is a source $S$ with $q$ symbols, and an instantaneous $r$-ary code $C$ satisfying $L(C)=H_r(S)$ if and only if $q\equiv 1 ...
3
votes
1answer
60 views

How to consider gravity as an entropic force?

Is it possible to consider gravity as an entropic force? What is mathematical relation between second law of thermodynamics and newton law of gravity?
3
votes
0answers
112 views

Can one define informational content of a mathematical expression?

At least in physicist's thinking, information, vaguely, is something that allows one to select a subset from a set. Say, a system can be in states A and B, we have done a measurement on it ...
2
votes
1answer
253 views

Graph Entropy - What is it?

I am having trouble getting some intuition as to what graph entropy measures. The definition that I have is that given a graph $G$, $H(G) = \min_{X,Y}I(X ;Y)$, where $X$ is a uniformly random vertex ...
0
votes
0answers
54 views

What is min-entropy?

Min-entropy is defined to be $ H_{\infty}$(x) = k if max$_x$ [Pr(X=x] ) = 2$^{-k}$ There are many research papers that use the terms: high min-entropy considerable min-entropy high computational ...
2
votes
1answer
93 views

Notion of Relative Entropy

I do not understand the notion of relative entropy. Relative Entropy. $D_{KL}(P||Q) = \sum_{i}^{}P(i)\log \frac{P(i)}{Q(i)}$. I try to get some intuition why it looks the way it looks. I see that it ...
5
votes
1answer
128 views

A tight lower bound for the entropy of the XOR of two random variables

Let $U$ be the uniform random variable over $n$-bit binary strings, and let $X$ be another random variable that is dependent on $U$ and ranges over $n$-bit binary strings. Assuming $I(X;U) \le ...
0
votes
0answers
32 views

Dependency of differential entropy on values of random variable

For a discrete random variable, entropy does not depend on the values-, but only probabilities of that random variable. Does this property also scale to the continuous case, when random variable X ...
1
vote
1answer
303 views

Maximum entropy joint distribution from marginals?

How does one find the maximum entropy joint distribution of two random variables X and Y given their marginal probability mass functions? I know: I have the marginals, meaning p(x) and p(y) are ...
0
votes
2answers
56 views

Entropy calculation

Let's say we have an unknown random variable whose entropy is $1.75$ Our job is to find minimum distributions for this random variable. What I wrote was: $$ p_1 \log_2\Big(\frac 1 {p_1}\Big) + \ldots ...
0
votes
1answer
75 views

How much information do you get if you draw a red card?

I'm trying to figure out what this question is asking and what it is I'm trying to calculate exactly. I'm told: You have cards 2-5 of each suit, except the 2 and 3 of the red cards. So 12 cards ...
0
votes
0answers
43 views

Creating feature vectors with a high degree of entropy from the actual training data for neural network?

Note: My upper level math skills are not in good shape and end around 1st semester calculus. I am looking for guidance that would help me find a known algorithm that does what I need in the open ...
0
votes
0answers
48 views

using entropy to calculate the relatedness of two columns in a database

There are two columns(x, y) in a database, I want to define the "relatedness" of the two columns. First i try to use I(x, y) (mutual information) to define the relatedness, then: date, ...
1
vote
3answers
423 views

Entropy of geometric random variable?

I am wondering how to derive the entropy of a geometric random variable? Or where I can find some proof/derivation? I tried to search online, but seems not much resources is available. Here is the ...
1
vote
1answer
97 views

Is mutual information transitive?

Suppose A, B and C are random variables. Given that the mutual information between A and B is very large and also the mutual information between B and C is very large, could we conclude that the ...