This tag is for [Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)) in Mathematics.

learn more… | top users | synonyms

0
votes
0answers
28 views

Dependency of differential entropy on values of random variable

For a discrete random variable, entropy does not depend on the values-, but only probabilities of that random variable. Does this property also scale to the continuous case, when random variable X ...
1
vote
1answer
246 views

Maximum entropy joint distribution from marginals?

How does one find the maximum entropy joint distribution of two random variables X and Y given their marginal probability mass functions? I know: I have the marginals, meaning p(x) and p(y) are ...
0
votes
2answers
51 views

Entropy calculation

Let's say we have an unknown random variable whose entropy is $1.75$ Our job is to find minimum distributions for this random variable. What I wrote was: $$ p_1 \log_2\Big(\frac 1 {p_1}\Big) + \ldots ...
0
votes
1answer
72 views

How much information do you get if you draw a red card?

I'm trying to figure out what this question is asking and what it is I'm trying to calculate exactly. I'm told: You have cards 2-5 of each suit, except the 2 and 3 of the red cards. So 12 cards ...
0
votes
0answers
34 views

Creating feature vectors with a high degree of entropy from the actual training data for neural network?

Note: My upper level math skills are not in good shape and end around 1st semester calculus. I am looking for guidance that would help me find a known algorithm that does what I need in the open ...
0
votes
0answers
45 views

using entropy to calculate the relatedness of two columns in a database

There are two columns(x, y) in a database, I want to define the "relatedness" of the two columns. First i try to use I(x, y) (mutual information) to define the relatedness, then: date, ...
1
vote
3answers
294 views

Entropy of geometric random variable?

I am wondering how to derive the entropy of a geometric random variable? Or where I can find some proof/derivation? I tried to search online, but seems not much resources is available. Here is the ...
1
vote
1answer
76 views

Is mutual information transitive?

Suppose A, B and C are random variables. Given that the mutual information between A and B is very large and also the mutual information between B and C is very large, could we conclude that the ...
1
vote
0answers
23 views

Best path for finding within a radius of x units from this point

Say i am standing at a point and knew there is one thing within a radius of x units from this point. What is best path to find that thing. Best can mean shortest, but the discussion can be more open. ...
2
votes
1answer
57 views

How are definitions of chaos related?

Chaotic systems can be defined in many ways. One definition is that the system has a positive Lyapunov exponent, that is, two trajectories starting near each other will diverge exponentially quickly. ...
6
votes
3answers
209 views

What is information theoretic entropy and its physical significance?

I have learned entropy in my information theory classes. The definition I got from text books was the average information content in a message sequence etc. But in one of the MIT videos related to ...
0
votes
0answers
54 views

Von Neumann Entropy Inequality

Suppose $\rho_1$ and $\rho_2$ are density matrices (and thus Hermitian, positive semi-definite matrices) and $\hat{w}$ is the solution of the following optimization problem $$ \hat{w} = ...
1
vote
0answers
36 views

Calculating entropy of Naive Bayes random variables

Suppose a Naive Bayes graphical model with binary random variables is given by $$P(y,x_1,x_2,...,x_n)=P(y)P(x_1|y)...P(x_n|y)$$ Attempting to calculate $I(x_1,...,x_n;y)$ raises the question: how can ...
0
votes
1answer
160 views

Bounded sets of isolated points in compact metric spaces

Context and definitions: Say $(X,d)$ is a compact metric space, with $f: X \rightarrow X$ continuous. For each $n \in \mathbb{N}$, the metric $d_{n}(x,y) = \max_{0 \leq k \leq ...
1
vote
1answer
196 views

Is maximizing the Shannon differential entropy equivalent to minimizing the predictability and/or minimizing the maximum density?

For a real-valued, 1-dimensional, continuous random variable X with density f(x), I am trying to determine if maximizing the Shannon differential entropy of f(x) is mathematically equivalent to ...
0
votes
1answer
139 views

Information theory entropy equation

I'm studying information theory, and working through this document. On page 17, it shows that, with the function that gets the entropy of a probability $I$ and a probability $p$, that $I(p^a) = a * ...
1
vote
0answers
36 views

Optimization problem in the Von Neumann Entropy

I have a constrainted optimization problem in the Von Neumann Entropy. In a CVX-like syntax the problem goes as follows: given variable $\mathtt{c(n)}$ $$\begin{align} \text{minimize} \qquad & ...
2
votes
0answers
121 views

Non-zero Conditional Differential Entropy between a random variable and a function of it

Let two continuous random variables, where the one is a function of the other: $X\, $ and $\, Y=g\left(X\right)$. Their mutual information is defined as ...
2
votes
1answer
64 views

Entropy and Shearer's Inequality

I have two questions both related to Shearer's Inequality: 1) When is equality attained in Shearer's Inequality? One trivial instance is when the random variables are all independent. Is this the ...
1
vote
1answer
95 views

von Neumann entropy and change of basis

The von Neumann entropy is defined as $S(\rho)=-Tr({\rho \ln \rho})$, where $\rho$ is density matrix. http://en.wikipedia.org/wiki/Von_Neumann_entropy In the above article it says: S(ρ) is ...
2
votes
1answer
72 views

Entropy and conditional probability given several variables

Following Calculating conditional entropy given two random variables I would like to figure out how to extend the entropy to several given variables: For two variables: $H(X|Y,Z) = -\sum ...
4
votes
1answer
49 views

Inequality on Shannon's entropy

Let $P$ be a set of probabilities s.t. $\sum_{p_i \in P} p_i = 1$. Moreover, let $H(P)$ the Shannon's entropy of the set of probabilities $P$: $$ H(P) = -\sum_{p_i \in P} p_i \log_2 p_i $$ I define ...
2
votes
1answer
88 views

Equality of sets when minimizing Shannon's Entropy

Let $P = \{p_1, \ldots, p_n\}$ be a set of probabilities, i.e., $0 \leq p_i \leq 1$. $P$ is such that $\sum_{p_i \in P} p_i = 1$. I have a set of actions $\mathcal{A} = \{a_1, \ldots, a_N\}$ that can ...
0
votes
1answer
108 views

What is the maximum entropy distribution of points on a sphere that has a fixed non-zero average cosine of the polar angle?

Suppose we have a unit vector in 3D space whose orientation has some unknown distribution $p(\theta,\phi)$. All we know about this distribution is the average value of $cos(\theta)$: ...
2
votes
1answer
74 views

Shannon's entropy in a set of probabilities

Let $P = p_1, \ldots, p_N$ be a set of probabilities (i.e., $0 \leq p_i \leq 1$). I can compute the Shannon's entropy as follows: $$ H(P) = -\sum_{i=1}^N p_i \log_2 p_i $$ Now, suppose I perform the ...
3
votes
3answers
252 views

Relationship between entropy and ergodicity

Are there any direct connections between entropy and ergodicity? For example, does knowing that $(X,\mathcal{S},\mu,T)$ is ergodic help in computing the entropy? I know that there are some indirect ...
3
votes
0answers
104 views

Reference for a transformation

Has the (Lebesgue-)ergodic transformation $T: \{0,1\}^{\mathbb{N}} \to \{0,1\}^{\mathbb{N}}$ defined by $T(x(0)x(1)x(2)\cdots) = x(1)x(3)x(5)\cdots$ been well-studied? If so, where? Does it have a ...
0
votes
0answers
156 views

How to compute the Shannon entropy for financial time series?

would someone be so kind the explain me how to compute the Shannon entropy for a real value time serie? As example, let's suppose time serie length = $100$. What are the steps I have to follow? ...
0
votes
0answers
24 views

Some doubts related to statistic entropy concept in ID3 machine learning algorithm

I am studying the statistic entropy concept used by ID3 machine learning algorithm For a domain exemplified by the learning set S (that is the set of the examples that I use to build a decision ...
1
vote
0answers
30 views

Entropy of a measure-preserving transformation

Let $T:X \rightarrow X$ be a measure-preserving transformation of the probability space $(X,B,\mu)$. ($X$ is a topological space). If $K\subset X$ is a compact and invariant set, show that ...
0
votes
1answer
75 views

Entropy Inequality

I have very hard time to prove the following inequality or to show a contradiction. $H(X_1,X_2,X_3) + H(X_1,X_2,X_4)+ H(X_1,X_3,X_4) + H(X_2,X_3,X_4) \leq 3(H(X_1,X_2) + H(X_3,X_4))$ The problem ...
2
votes
1answer
86 views

Properties of Entropy

When someone writes $H(X_1, X_2, X_3) = H(X_1) + H(X_2\mid X_1) + H(X_3\mid X_2, X_1)$, how should that last term be interpreted/read? As the joint entropy between 2 variables where variable 1 is ...
1
vote
1answer
273 views

How Entropy scales with sample size

For a discrete probability distribution, the entropy is defined as: $$H(p) = \sum_i p(x_i) \log(p(x_i))$$ I'm trying to use the entropy as a measure of how "flat / noisy" vs. "peaked" a distribution ...
0
votes
1answer
39 views

Help with a solution about entropy!

I'm triying to solve this question question, but I don't understand why $$ -\log_2 (2\pi n pq)/(2n) $$ transforms into $$O(\ln n/ n)$$ can you help me please :( !!
0
votes
1answer
85 views

Solving exponential equations for maxent with opensource applications or coding

I am working on maximum entropy. And I have a problem solving some equations. Look at these 2 equations: $\frac{(2*e^{-a*2-b*2^2}) +(6*e^{-a*6-b*6^2}) +(10*e^{-a*10-b*10^2}) ...
1
vote
1answer
51 views

Constructing Distribution By Coin Flipping

I am interested in any example of construction distribution by coin flipping. Actually I want to show the process of construction a random variable $X$ with distribution $(p_1,...,p_n)$ by coin ...
0
votes
0answers
63 views

Intregral of exponential of Shannon Entropy Function

Here I am going to ask a similar question as rde asked , that is what is the integral of exponential of entropy function. That is what is the value of $F[H(x)]=\int_{-1}^{+1} e^{ikH(f(x^2))} dx$ ...
4
votes
0answers
82 views

A combinatorial problem.

Let be $(X, \mathbb{A}, \mu)$ a measure space, a partition of $ X $ is a disjoint family $\xi=\{P_1,\ldots,P_k \}$ of measurable sets such tath $\bigcup P_i=X\pmod0).$ If $\xi=\{P_1,\ldots,P_k ...
4
votes
2answers
85 views

Are there logarithm functions for arbitrary rings?

The logarithm function for $\mathbb{R}$ satisfies $\log xy = \log x + \log y$ whenever both $\log x$ and $\log y$ are defined. Are their conditions for a ring $R$ which guarantee the existence of a ...
1
vote
1answer
65 views

Entropy vs predictability vs encodability

Imagine there's a guessing game where a series of binary symbols are presented and a human must decide quickly if the symbol is the same as the previous or different. There's a property of the ...
0
votes
0answers
132 views

Replace a continuous probability distribution with a discrete one

Say one wants to fit a curve $f(x)$ to a set of noisy data points $(x_i, y_i)$. If the error for each point $y_i$ is assumed to be normally distributed with variance $\sigma_i^2$, one wants to find ...
2
votes
1answer
45 views

Shannon inequalities

I have some difficulties in showing the relationship between mutual information $I(X; Y |Z)$ and $I(X; Y)$? What is larger?
2
votes
0answers
52 views

Computing Relative entropy?

I am doing a project for my CS class and I was wondering if the following would work. I have 50 different people who have rated the same 50 books. The rating system is as follows: negative 5 = hate ...
1
vote
0answers
59 views

convergence of discrete random variables with finite entropy

Let $Z$ be the set of discrete random variables on some probability space. Define the quantity $d(X_1,X_2)=h(X_1 \mid X_2)+h(X_2 \mid X_1)$ between two random variables $X_1, X_2 \in Z$. For $X \in Z$ ...
3
votes
0answers
87 views

Which takes more energy: Shuffling a sorted deck or sorting a shuffled one?

You have an array of length $n$ containing $n$ distinct elements. You have access to a comparator on the elements (a black-box function that takes $a$ and $b$ and returns true if $a < b$, false ...
1
vote
0answers
60 views

convexity of the product of two entropy-like functions

Consider the functions $T_p(q)= \sum_i q_i^p$, where p>1 and q is a finite-dimensional vector satisfying $\sum_i q_i = 1, q_i >0$ (ie, a probability mass function). In information-theoretic terms, ...
3
votes
1answer
36 views

Do the pth powers of $p$-norms define the same partial ordering on the set of all probability distributions for all $p>1$?

Consider the $p$-th power of the Schatten $p$-norm $||q||_p$ of a probability distribution $q$ , ie, the function $\sum_j q_j^p$, where $\sum_j q_j = 1$ and $q_j \geq 0$. For fixed $q$ and $p>1$ ...
0
votes
1answer
43 views

i.i.d binary random variable question

Suppose there are i.i.d. binary random variables $X_i \sim X$ with distribution $P(X=1) = 0.75$ and $P(X=0) = 0.25$ i) For $n=5$ and $e=0.1$, which sequences fall in the typical set $A_e^n$? What is ...
1
vote
0answers
88 views

Why is the negative entropy Lipschitz with respect to the $1$-norm (Over)?

Let $\left\|x \right\| = \sum_{i=1}^{i=n}\left|x^i\right|$ and $d\left(x\right)=\sum_{i=1}^{i=n}x^i\ln x^i$ where $x\in R^n $ and $ \sum_{i=1}^{i=n}x^i=1$ How to prove: For all $x, x'$, $$\left| ...
0
votes
1answer
155 views

Approximating probability of success of Bernoulli trials using Kullback–Leibler divergence

In "Probabilistic Graphical Models" book by Daphne Koller and Nir Friedman they have the following approximation of probability of r successful outcomes of N Bernoulli trials: $P(S_N=r)\approx ...