Tag Info

New answers tagged

0

This is not a definition of topological entropy, but rather a version of a result called the "variational principle". Can;t remember who it is due to. This result states that topological entropy of a map T coincides with its largest Kolmogorov-Sinai entropy, maximised over all T-invariant distributions. Note that the K-S entropy of T is basically the same as ...


0

This is not an answer but it is too long for a comment. Considering $$S(k)=\sum _{i=1}^k \frac{(i-1)! \binom{k}{i}^2}{\binom{\frac{1}{2} k(k-1) }{i}}$$ I have not been able to establish asymptotics but numerical simulations show that $\log\big(S(k)\big)$ varies almost linearly with $k$. Based on values computed for $k=100,200,\cdots,2000$ an empirical ...


0

The most natural (and almost trivial) way to estimate (not calculate) the probabilities is just counting: $$\hat{p_i}=\frac{n_i}{N}$$ where $p_i$ is the probabilty of symbol $i$, $\hat{p_i}$ its estimator, $n_i$ the counting of ocurrences of symbol $i$, and $n$ the total of samples. Then you plug this estimator into the entropy formula. However, this might ...


1

Your computation (actually, estimation) of the entropy in terms of a sequence, implicitly assumes that the succesive symbols are independent (and the source is stationary, of course). However, it seems intuitively true that the second sequence is more "random" than the first sequence and hence should, in some sense, have higher entropy. Your ...


2

By rescaling the bins, you are changing the number of internal states, which shifts the origin of the entropy scale. That's actually the exact same problem classical thermodynamics has that only gets resolved in quantum theory, where the number of microscopic states is limited by the quantization of the phase space. When the absolute entropy scale is set, ...


2

Yes, it is just the chain rule for entropy. The chain rule for entropy is Theorem 2.2.1 in Cover and Thomas. $$H(X,Y) = H(X) + H(Y|X)$$ You can use the chain rule when you condition on another random variable $Z$. In this case, you get $$H(X,Y|Z) = H(X|Z) + H(Y|X,Z)$$ The proof is word for word the same as the proof of the original chain rule. In this case, ...


1

Taking a quick look at the definition of the conditional entropy, given by, $$H(X|Y ) = \sum_{y } p(y)H(X|Y = y),$$ we can say that $H(X|Y = y)$ is a random variable, but $H(X|Y)$ is a constant. Note that $H(X|Y)$ is the expected value of $H(X|Y = y)$ over $y$.


1

For a very light introduction, see http://www.cybertester.com/data/wavelet.pdf The general idea is that a family of wavelets form a basis for a space of functions, where the basis functions are bounded in both time and frequency. As with any transformation to a different basis, we have the possibility that only a (small) finite number of coefficients are ...


2

The entropy function is strictly concave, has unique maximum $\log 3$ at the center, and its restriction to the boundary segments is also strictly concave with unique maxima at $(0, 1/2, 1/2)$ and the permutations, the value at each of these 3 points being $\log 2$. Thus the boundary states (where at least one $p_i$ is $0$) have possible entropy values ...


1

From definition of Markovity in $p(x,y,z) = p(x)p(y|x)p(z|y)$, one can prove that $p(x,z|y) = p(x|y)p(z|y)$. This will give the following as another definition of Markovity: Past and future are conditionally independent given the present. With this, we can see that, if $X \rightarrow Y \rightarrow Z$, then $I(X; Z | Y)=0$. Therefore we have: $$I(X ; ...


0

Define a random vector $Z$ which is a result of concatenating $X$ and $Y$. $H(X,Y) := H(Z)$.



Top 50 recent answers are included