This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

3
votes
1answer
134 views

What does SVD Entropy Capture?

Looking at different definitions and types of Entropy, I run into the concept of SVD Entropy, which is defined as explained below. What is the intuition behind the SVD spectrum? What do different ...
2
votes
1answer
50 views

How to prove the function is logarithmic with coefficience

I am given a set of properties for an unknown function $f(x)$. In particular, not constantly zero, not negative, additive and continues for any $x$ between 0 and 1. I am asked to show equivalence ...
2
votes
1answer
75 views

Why $p$ factor in $p \log p$ in the entropy formula?

It is explained that we have log for additivity of information in the entropy formula. But, why is the $p$ factor? It is redundant, since we already have it in the $\log p$!
2
votes
0answers
135 views

How many code words if average code length equals entropy

I've been given a proof of the following: If $q\geq2$, then there is a source $S$ with $q$ symbols, and an instantaneous $r$-ary code $C$ satisfying $L(C)=H_r(S)$ if and only if $q\equiv 1 ...
3
votes
1answer
60 views

How to consider gravity as an entropic force?

Is it possible to consider gravity as an entropic force? What is mathematical relation between second law of thermodynamics and newton law of gravity?
3
votes
0answers
115 views

Can one define informational content of a mathematical expression?

At least in physicist's thinking, information, vaguely, is something that allows one to select a subset from a set. Say, a system can be in states A and B, we have done a measurement on it ...
2
votes
1answer
286 views

Graph Entropy - What is it?

I am having trouble getting some intuition as to what graph entropy measures. The definition that I have is that given a graph $G$, $H(G) = \min_{X,Y}I(X ;Y)$, where $X$ is a uniformly random vertex ...
2
votes
1answer
98 views

Notion of Relative Entropy

I do not understand the notion of relative entropy. Relative Entropy. $D_{KL}(P||Q) = \sum_{i}^{}P(i)\log \frac{P(i)}{Q(i)}$. I try to get some intuition why it looks the way it looks. I see that it ...
5
votes
1answer
147 views

A tight lower bound for the entropy of the XOR of two random variables

Let $U$ be the uniform random variable over $n$-bit binary strings, and let $X$ be another random variable that is dependent on $U$ and ranges over $n$-bit binary strings. Assuming $I(X;U) \le ...
1
vote
1answer
344 views

Maximum entropy joint distribution from marginals?

How does one find the maximum entropy joint distribution of two random variables X and Y given their marginal probability mass functions? I know: I have the marginals, meaning p(x) and p(y) are ...
0
votes
2answers
63 views

Entropy calculation

Let's say we have an unknown random variable whose entropy is $1.75$ Our job is to find minimum distributions for this random variable. What I wrote was: $$ p_1 \log_2\Big(\frac 1 {p_1}\Big) + \ldots ...
0
votes
1answer
79 views

How much information do you get if you draw a red card?

I'm trying to figure out what this question is asking and what it is I'm trying to calculate exactly. I'm told: You have cards 2-5 of each suit, except the 2 and 3 of the red cards. So 12 cards ...
0
votes
0answers
48 views

using entropy to calculate the relatedness of two columns in a database

There are two columns(x, y) in a database, I want to define the "relatedness" of the two columns. First i try to use I(x, y) (mutual information) to define the relatedness, then: date, ...
1
vote
3answers
582 views

Entropy of geometric random variable?

I am wondering how to derive the entropy of a geometric random variable? Or where I can find some proof/derivation? I tried to search online, but seems not much resources is available. Here is the ...
1
vote
1answer
101 views

Is mutual information transitive?

Suppose A, B and C are random variables. Given that the mutual information between A and B is very large and also the mutual information between B and C is very large, could we conclude that the ...
1
vote
0answers
24 views

Best path for finding within a radius of x units from this point

Say i am standing at a point and knew there is one thing within a radius of x units from this point. What is best path to find that thing. Best can mean shortest, but the discussion can be more open. ...
2
votes
1answer
66 views

How are definitions of chaos related?

Chaotic systems can be defined in many ways. One definition is that the system has a positive Lyapunov exponent, that is, two trajectories starting near each other will diverge exponentially quickly. ...
6
votes
3answers
293 views

What is information theoretic entropy and its physical significance?

I have learned entropy in my information theory classes. The definition I got from text books was the average information content in a message sequence etc. But in one of the MIT videos related to ...
1
vote
0answers
40 views

Calculating entropy of Naive Bayes random variables

Suppose a Naive Bayes graphical model with binary random variables is given by $$P(y,x_1,x_2,...,x_n)=P(y)P(x_1|y)...P(x_n|y)$$ Attempting to calculate $I(x_1,...,x_n;y)$ raises the question: how can ...
0
votes
1answer
190 views

Bounded sets of isolated points in compact metric spaces

Context and definitions: Say $(X,d)$ is a compact metric space, with $f: X \rightarrow X$ continuous. For each $n \in \mathbb{N}$, the metric $d_{n}(x,y) = \max_{0 \leq k \leq ...
1
vote
1answer
228 views

Is maximizing the Shannon differential entropy equivalent to minimizing the predictability and/or minimizing the maximum density?

For a real-valued, 1-dimensional, continuous random variable X with density f(x), I am trying to determine if maximizing the Shannon differential entropy of f(x) is mathematically equivalent to ...
0
votes
1answer
188 views

Information theory entropy equation

I'm studying information theory, and working through this document. On page 17, it shows that, with the function that gets the entropy of a probability $I$ and a probability $p$, that $I(p^a) = a * ...
1
vote
0answers
49 views

Optimization problem in the Von Neumann Entropy

I have a constrainted optimization problem in the Von Neumann Entropy. In a CVX-like syntax the problem goes as follows: given variable $\mathtt{c(n)}$ $$\begin{align} \text{minimize} \qquad & ...
3
votes
0answers
150 views

Non-zero Conditional Differential Entropy between a random variable and a function of it

Let two continuous random variables, where the one is a function of the other: $X\, $ and $\, Y=g\left(X\right)$. Their mutual information is defined as ...
2
votes
1answer
85 views

Entropy and Shearer's Inequality

I have two questions both related to Shearer's Inequality: 1) When is equality attained in Shearer's Inequality? One trivial instance is when the random variables are all independent. Is this the ...
1
vote
1answer
122 views

von Neumann entropy and change of basis

The von Neumann entropy is defined as $S(\rho)=-Tr({\rho \ln \rho})$, where $\rho$ is density matrix. http://en.wikipedia.org/wiki/Von_Neumann_entropy In the above article it says: S(ρ) is ...
2
votes
1answer
81 views

Entropy and conditional probability given several variables

Following Calculating conditional entropy given two random variables I would like to figure out how to extend the entropy to several given variables: For two variables: $H(X|Y,Z) = -\sum ...
4
votes
1answer
54 views

Inequality on Shannon's entropy

Let $P$ be a set of probabilities s.t. $\sum_{p_i \in P} p_i = 1$. Moreover, let $H(P)$ the Shannon's entropy of the set of probabilities $P$: $$ H(P) = -\sum_{p_i \in P} p_i \log_2 p_i $$ I define ...
2
votes
1answer
99 views

Equality of sets when minimizing Shannon's Entropy

Let $P = \{p_1, \ldots, p_n\}$ be a set of probabilities, i.e., $0 \leq p_i \leq 1$. $P$ is such that $\sum_{p_i \in P} p_i = 1$. I have a set of actions $\mathcal{A} = \{a_1, \ldots, a_N\}$ that can ...
0
votes
1answer
153 views

What is the maximum entropy distribution of points on a sphere that has a fixed non-zero average cosine of the polar angle?

Suppose we have a unit vector in 3D space whose orientation has some unknown distribution $p(\theta,\phi)$. All we know about this distribution is the average value of $cos(\theta)$: ...
2
votes
1answer
82 views

Shannon's entropy in a set of probabilities

Let $P = p_1, \ldots, p_N$ be a set of probabilities (i.e., $0 \leq p_i \leq 1$). I can compute the Shannon's entropy as follows: $$ H(P) = -\sum_{i=1}^N p_i \log_2 p_i $$ Now, suppose I perform the ...
3
votes
3answers
322 views

Relationship between entropy and ergodicity

Are there any direct connections between entropy and ergodicity? For example, does knowing that $(X,\mathcal{S},\mu,T)$ is ergodic help in computing the entropy? I know that there are some indirect ...
3
votes
0answers
107 views

Reference for a transformation

Has the (Lebesgue-)ergodic transformation $T: \{0,1\}^{\mathbb{N}} \to \{0,1\}^{\mathbb{N}}$ defined by $T(x(0)x(1)x(2)\cdots) = x(1)x(3)x(5)\cdots$ been well-studied? If so, where? Does it have a ...
0
votes
0answers
218 views

How to compute the Shannon entropy for financial time series?

would someone be so kind the explain me how to compute the Shannon entropy for a real value time serie? As example, let's suppose time serie length = $100$. What are the steps I have to follow? ...
1
vote
0answers
35 views

Entropy of a measure-preserving transformation

Let $T:X \rightarrow X$ be a measure-preserving transformation of the probability space $(X,B,\mu)$. ($X$ is a topological space). If $K\subset X$ is a compact and invariant set, show that ...
0
votes
1answer
86 views

Entropy Inequality

I have very hard time to prove the following inequality or to show a contradiction. $H(X_1,X_2,X_3) + H(X_1,X_2,X_4)+ H(X_1,X_3,X_4) + H(X_2,X_3,X_4) \leq 3(H(X_1,X_2) + H(X_3,X_4))$ The problem ...
2
votes
1answer
90 views

Properties of Entropy

When someone writes $H(X_1, X_2, X_3) = H(X_1) + H(X_2\mid X_1) + H(X_3\mid X_2, X_1)$, how should that last term be interpreted/read? As the joint entropy between 2 variables where variable 1 is ...
3
votes
2answers
530 views

How Entropy scales with sample size

For a discrete probability distribution, the entropy is defined as: $$H(p) = \sum_i p(x_i) \log(p(x_i))$$ I'm trying to use the entropy as a measure of how "flat / noisy" vs. "peaked" a distribution ...
0
votes
1answer
44 views

Help with a solution about entropy!

I'm triying to solve this question question, but I don't understand why $$ -\log_2 (2\pi n pq)/(2n) $$ transforms into $$O(\ln n/ n)$$ can you help me please :( !!
0
votes
1answer
138 views

Solving exponential equations for maxent with opensource applications or coding

I am working on maximum entropy. And I have a problem solving some equations. Look at these 2 equations: $\frac{(2*e^{-a*2-b*2^2}) +(6*e^{-a*6-b*6^2}) +(10*e^{-a*10-b*10^2}) ...
1
vote
1answer
55 views

Constructing Distribution By Coin Flipping

I am interested in any example of construction distribution by coin flipping. Actually I want to show the process of construction a random variable $X$ with distribution $(p_1,...,p_n)$ by coin ...
0
votes
0answers
72 views

Intregral of exponential of Shannon Entropy Function

Here I am going to ask a similar question as rde asked , that is what is the integral of exponential of entropy function. That is what is the value of $F[H(x)]=\int_{-1}^{+1} e^{ikH(f(x^2))} dx$ ...
4
votes
0answers
87 views

A combinatorial problem.

Let be $(X, \mathbb{A}, \mu)$ a measure space, a partition of $ X $ is a disjoint family $\xi=\{P_1,\ldots,P_k \}$ of measurable sets such tath $\bigcup P_i=X\pmod0).$ If $\xi=\{P_1,\ldots,P_k ...
4
votes
2answers
89 views

Are there logarithm functions for arbitrary rings?

The logarithm function for $\mathbb{R}$ satisfies $\log xy = \log x + \log y$ whenever both $\log x$ and $\log y$ are defined. Are their conditions for a ring $R$ which guarantee the existence of a ...
1
vote
1answer
79 views

Entropy vs predictability vs encodability

Imagine there's a guessing game where a series of binary symbols are presented and a human must decide quickly if the symbol is the same as the previous or different. There's a property of the ...
0
votes
0answers
140 views

Replace a continuous probability distribution with a discrete one

Say one wants to fit a curve $f(x)$ to a set of noisy data points $(x_i, y_i)$. If the error for each point $y_i$ is assumed to be normally distributed with variance $\sigma_i^2$, one wants to find ...
2
votes
1answer
49 views

Shannon inequalities

I have some difficulties in showing the relationship between mutual information $I(X; Y |Z)$ and $I(X; Y)$? What is larger?
2
votes
0answers
56 views

Computing Relative entropy?

I am doing a project for my CS class and I was wondering if the following would work. I have 50 different people who have rated the same 50 books. The rating system is as follows: negative 5 = hate ...
1
vote
0answers
61 views

convergence of discrete random variables with finite entropy

Let $Z$ be the set of discrete random variables on some probability space. Define the quantity $d(X_1,X_2)=h(X_1 \mid X_2)+h(X_2 \mid X_1)$ between two random variables $X_1, X_2 \in Z$. For $X \in Z$ ...
3
votes
0answers
88 views

Which takes more energy: Shuffling a sorted deck or sorting a shuffled one?

You have an array of length $n$ containing $n$ distinct elements. You have access to a comparator on the elements (a black-box function that takes $a$ and $b$ and returns true if $a < b$, false ...