This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

1
vote
1answer
33 views

Proving the monotonicity of a recurrence.

Define the following recurrence for $n = 1, 2, \cdots$ $T(n) = ( 1 - \operatorname{H}(\frac{1 - P^{\frac{1}{n}}}{2}))^n$ where $0 < P < 1$ is a constant, function $\operatorname{H}(\cdot)$ is ...
0
votes
2answers
48 views

Properties of joint entropy

I'm trying to show the following, but am stuck For discrete random variables $X$ and $Y$, show $$H(X,Y)\geq \max\{H(X),H(Y)\}$$ where $H$ represents entropy
1
vote
2answers
48 views

Entropy of a distribution over strings

Suppose for some parameter $d$, we choose a string from the Hamming cube ($\{0,1\}^d$) by setting each bit to be $0$ with probability $p$ and $1$ with probability $1-p$. What is the entropy of this ...
1
vote
1answer
59 views

Kullback–Leibler divergence in bits

Well known formula of KL divergence when we have a discrete probability distributions. $$D_{KL}(P \parallel Q)=\sum\limits_i \ln \left(\frac{P(i)}{Q(i)}\right) P(i)$$ Can someone explain why the ...
1
vote
1answer
56 views

Calculate entropy of modified base32 hash

I am trying to develop a scheme for generating unique (probable within bounds) ids in a distributed application. I want the id to be easily remembered, easily spoken, and easily read. I chose base32 ...
1
vote
0answers
76 views

what is relative entropy between to random binary string with length of $L_1$ & $L_2$?

I want calculate relative entropy between two strings of binary such as: $L_1:11000100011101001$ $L_2:00101110110111001$ It is primarily when the lengths of two strings is same and in general when ...
0
votes
1answer
27 views

$Y$ is a function of $X$: making an inference based on the markovity of $ X$

In the information theory book by Cover and Thomas it is written: if $X$ is markov and $Y$ is a function of $X$ then: ...
0
votes
2answers
52 views

Can infinite random sequences be asymptotically compressed?

A number $0.5<p<1$ is chosen at random and given to two people A and B whom are allowed to communicate before beeing separated. A is then given a sequence S of N random bits where each bit has ...
1
vote
1answer
89 views

Maximising Entropy of Random Variable taking Positive Integer Values

A random variable $X$ takes positive integer values and $E[X]=6$. What distribution of the random variable $X$ maximises the entropy $H(X)$? What if $X$ can only take a finite number of values? ...
3
votes
1answer
155 views

What does SVD Entropy Capture?

Looking at different definitions and types of Entropy, I run into the concept of SVD Entropy, which is defined as explained below. What is the intuition behind the SVD spectrum? What do different ...
2
votes
1answer
55 views

How to prove the function is logarithmic with coefficience

I am given a set of properties for an unknown function $f(x)$. In particular, not constantly zero, not negative, additive and continues for any $x$ between 0 and 1. I am asked to show equivalence ...
2
votes
1answer
77 views

Why $p$ factor in $p \log p$ in the entropy formula?

It is explained that we have log for additivity of information in the entropy formula. But, why is the $p$ factor? It is redundant, since we already have it in the $\log p$!
2
votes
0answers
144 views

How many code words if average code length equals entropy

I've been given a proof of the following: If $q\geq2$, then there is a source $S$ with $q$ symbols, and an instantaneous $r$-ary code $C$ satisfying $L(C)=H_r(S)$ if and only if $q\equiv 1 ...
3
votes
1answer
60 views

How to consider gravity as an entropic force?

Is it possible to consider gravity as an entropic force? What is mathematical relation between second law of thermodynamics and newton law of gravity?
3
votes
0answers
115 views

Can one define informational content of a mathematical expression?

At least in physicist's thinking, information, vaguely, is something that allows one to select a subset from a set. Say, a system can be in states A and B, we have done a measurement on it ...
2
votes
1answer
308 views

Graph Entropy - What is it?

I am having trouble getting some intuition as to what graph entropy measures. The definition that I have is that given a graph $G$, $H(G) = \min_{X,Y}I(X ;Y)$, where $X$ is a uniformly random vertex ...
2
votes
1answer
105 views

Notion of Relative Entropy

I do not understand the notion of relative entropy. Relative Entropy. $D_{KL}(P||Q) = \sum_{i}^{}P(i)\log \frac{P(i)}{Q(i)}$. I try to get some intuition why it looks the way it looks. I see that it ...
5
votes
1answer
159 views

A tight lower bound for the entropy of the XOR of two random variables

Let $U$ be the uniform random variable over $n$-bit binary strings, and let $X$ be another random variable that is dependent on $U$ and ranges over $n$-bit binary strings. Assuming $I(X;U) \le ...
1
vote
1answer
366 views

Maximum entropy joint distribution from marginals?

How does one find the maximum entropy joint distribution of two random variables X and Y given their marginal probability mass functions? I know: I have the marginals, meaning p(x) and p(y) are ...
0
votes
2answers
65 views

Entropy calculation

Let's say we have an unknown random variable whose entropy is $1.75$ Our job is to find minimum distributions for this random variable. What I wrote was: $$ p_1 \log_2\Big(\frac 1 {p_1}\Big) + \ldots ...
0
votes
1answer
80 views

How much information do you get if you draw a red card?

I'm trying to figure out what this question is asking and what it is I'm trying to calculate exactly. I'm told: You have cards 2-5 of each suit, except the 2 and 3 of the red cards. So 12 cards ...
0
votes
0answers
49 views

using entropy to calculate the relatedness of two columns in a database

There are two columns(x, y) in a database, I want to define the "relatedness" of the two columns. First i try to use I(x, y) (mutual information) to define the relatedness, then: date, ...
1
vote
3answers
640 views

Entropy of geometric random variable?

I am wondering how to derive the entropy of a geometric random variable? Or where I can find some proof/derivation? I tried to search online, but seems not much resources is available. Here is the ...
1
vote
1answer
108 views

Is mutual information transitive?

Suppose A, B and C are random variables. Given that the mutual information between A and B is very large and also the mutual information between B and C is very large, could we conclude that the ...
1
vote
0answers
24 views

Best path for finding within a radius of x units from this point

Say i am standing at a point and knew there is one thing within a radius of x units from this point. What is best path to find that thing. Best can mean shortest, but the discussion can be more open. ...
2
votes
1answer
74 views

How are definitions of chaos related?

Chaotic systems can be defined in many ways. One definition is that the system has a positive Lyapunov exponent, that is, two trajectories starting near each other will diverge exponentially quickly. ...
6
votes
3answers
315 views

What is information theoretic entropy and its physical significance?

I have learned entropy in my information theory classes. The definition I got from text books was the average information content in a message sequence etc. But in one of the MIT videos related to ...
1
vote
0answers
40 views

Calculating entropy of Naive Bayes random variables

Suppose a Naive Bayes graphical model with binary random variables is given by $$P(y,x_1,x_2,...,x_n)=P(y)P(x_1|y)...P(x_n|y)$$ Attempting to calculate $I(x_1,...,x_n;y)$ raises the question: how can ...
0
votes
1answer
195 views

Bounded sets of isolated points in compact metric spaces

Context and definitions: Say $(X,d)$ is a compact metric space, with $f: X \rightarrow X$ continuous. For each $n \in \mathbb{N}$, the metric $d_{n}(x,y) = \max_{0 \leq k \leq ...
1
vote
1answer
242 views

Is maximizing the Shannon differential entropy equivalent to minimizing the predictability and/or minimizing the maximum density?

For a real-valued, 1-dimensional, continuous random variable X with density f(x), I am trying to determine if maximizing the Shannon differential entropy of f(x) is mathematically equivalent to ...
0
votes
1answer
188 views

Information theory entropy equation

I'm studying information theory, and working through this document. On page 17, it shows that, with the function that gets the entropy of a probability $I$ and a probability $p$, that $I(p^a) = a * ...
1
vote
0answers
49 views

Optimization problem in the Von Neumann Entropy

I have a constrainted optimization problem in the Von Neumann Entropy. In a CVX-like syntax the problem goes as follows: given variable $\mathtt{c(n)}$ $$\begin{align} \text{minimize} \qquad & ...
3
votes
0answers
157 views

Non-zero Conditional Differential Entropy between a random variable and a function of it

Let two continuous random variables, where the one is a function of the other: $X\, $ and $\, Y=g\left(X\right)$. Their mutual information is defined as ...
2
votes
1answer
91 views

Entropy and Shearer's Inequality

I have two questions both related to Shearer's Inequality: 1) When is equality attained in Shearer's Inequality? One trivial instance is when the random variables are all independent. Is this the ...
1
vote
1answer
132 views

von Neumann entropy and change of basis

The von Neumann entropy is defined as $S(\rho)=-Tr({\rho \ln \rho})$, where $\rho$ is density matrix. http://en.wikipedia.org/wiki/Von_Neumann_entropy In the above article it says: S(ρ) is ...
2
votes
1answer
82 views

Entropy and conditional probability given several variables

Following Calculating conditional entropy given two random variables I would like to figure out how to extend the entropy to several given variables: For two variables: $H(X|Y,Z) = -\sum ...
4
votes
1answer
54 views

Inequality on Shannon's entropy

Let $P$ be a set of probabilities s.t. $\sum_{p_i \in P} p_i = 1$. Moreover, let $H(P)$ the Shannon's entropy of the set of probabilities $P$: $$ H(P) = -\sum_{p_i \in P} p_i \log_2 p_i $$ I define ...
2
votes
1answer
100 views

Equality of sets when minimizing Shannon's Entropy

Let $P = \{p_1, \ldots, p_n\}$ be a set of probabilities, i.e., $0 \leq p_i \leq 1$. $P$ is such that $\sum_{p_i \in P} p_i = 1$. I have a set of actions $\mathcal{A} = \{a_1, \ldots, a_N\}$ that can ...
0
votes
1answer
161 views

What is the maximum entropy distribution of points on a sphere that has a fixed non-zero average cosine of the polar angle?

Suppose we have a unit vector in 3D space whose orientation has some unknown distribution $p(\theta,\phi)$. All we know about this distribution is the average value of $cos(\theta)$: ...
2
votes
1answer
84 views

Shannon's entropy in a set of probabilities

Let $P = p_1, \ldots, p_N$ be a set of probabilities (i.e., $0 \leq p_i \leq 1$). I can compute the Shannon's entropy as follows: $$ H(P) = -\sum_{i=1}^N p_i \log_2 p_i $$ Now, suppose I perform the ...
3
votes
3answers
334 views

Relationship between entropy and ergodicity

Are there any direct connections between entropy and ergodicity? For example, does knowing that $(X,\mathcal{S},\mu,T)$ is ergodic help in computing the entropy? I know that there are some indirect ...
3
votes
0answers
107 views

Reference for a transformation

Has the (Lebesgue-)ergodic transformation $T: \{0,1\}^{\mathbb{N}} \to \{0,1\}^{\mathbb{N}}$ defined by $T(x(0)x(1)x(2)\cdots) = x(1)x(3)x(5)\cdots$ been well-studied? If so, where? Does it have a ...
0
votes
0answers
225 views

How to compute the Shannon entropy for financial time series?

would someone be so kind the explain me how to compute the Shannon entropy for a real value time serie? As example, let's suppose time serie length = $100$. What are the steps I have to follow? ...
1
vote
0answers
39 views

Entropy of a measure-preserving transformation

Let $T:X \rightarrow X$ be a measure-preserving transformation of the probability space $(X,B,\mu)$. ($X$ is a topological space). If $K\subset X$ is a compact and invariant set, show that ...
0
votes
1answer
90 views

Entropy Inequality

I have very hard time to prove the following inequality or to show a contradiction. $H(X_1,X_2,X_3) + H(X_1,X_2,X_4)+ H(X_1,X_3,X_4) + H(X_2,X_3,X_4) \leq 3(H(X_1,X_2) + H(X_3,X_4))$ The problem ...
2
votes
1answer
93 views

Properties of Entropy

When someone writes $H(X_1, X_2, X_3) = H(X_1) + H(X_2\mid X_1) + H(X_3\mid X_2, X_1)$, how should that last term be interpreted/read? As the joint entropy between 2 variables where variable 1 is ...
3
votes
2answers
648 views

How Entropy scales with sample size

For a discrete probability distribution, the entropy is defined as: $$H(p) = \sum_i p(x_i) \log(p(x_i))$$ I'm trying to use the entropy as a measure of how "flat / noisy" vs. "peaked" a distribution ...
0
votes
1answer
46 views

Help with a solution about entropy!

I'm triying to solve this question question, but I don't understand why $$ -\log_2 (2\pi n pq)/(2n) $$ transforms into $$O(\ln n/ n)$$ can you help me please :( !!
0
votes
1answer
146 views

Solving exponential equations for maxent with opensource applications or coding

I am working on maximum entropy. And I have a problem solving some equations. Look at these 2 equations: $\frac{(2*e^{-a*2-b*2^2}) +(6*e^{-a*6-b*6^2}) +(10*e^{-a*10-b*10^2}) ...
1
vote
1answer
55 views

Constructing Distribution By Coin Flipping

I am interested in any example of construction distribution by coin flipping. Actually I want to show the process of construction a random variable $X$ with distribution $(p_1,...,p_n)$ by coin ...