Tag Info

New answers tagged

2

Topological entropy basically counts the (exponential) number of distinct orbits generated by the dynamical system, up to some 'resolution' and orbit length. The $\epsilon$ limit is basically taking your resolution to 0, and $n$ limit is taking the orbit length to infinity.


0

To answer the first question, it seems this procedure is called the Serial Approximate Entropy Test, according to section 2.12 of the NIST Statistical Test Suite for the Validation of Random Number Generators and Pseudo Random Number Generators.


1

In Shannon Entropy, MUST the probability be based solely on the sequence itself or can the probabilities be predetermined Rather on the contrary (if I understand you right): the probabilities must be predetermined. More precisely: the Shannon entropy is defined in terms of a probabilistic model, it assumes that the probabilities are known. Hence, it ...


0

@ Srivatsan Balakrishnan , you write anything. We assume that $A$ has no eigenvalues in $(-\infty,0)]$ and $\log(.)$ is the principal logarithm. The Taylor formula is: $\log(A+H)=\log(A)+Df_A(H)+\dfrac{1}{2}D^2f_A(H,H)+o(||H||^2)$. Now we calculate the first and second derivatives of the matrix function $f(A)=\log(A)$: ...


0

We assume that $A(x)$ has no eigenvalues in $(-\infty,0]$ and $\log$ denotes the principal logarithm. Since $g(A)=\log(A)$ is a matrix function, (E): $\log(A)$ is a polynomial in $A$. Let $P$ be a polynomial and $f:x\rightarrow tr(P(A)\log(A))$. According to (E), $f'(x)=tr(P'(A)\log(A)A')+tr(P(A)Dg_A(A'))$. Now ...


2

It is neither convex nor concave. You can work it out using Bernoulli random variables. Not convex: 1) Let $X$ and $Y$ be i.i.d. Bernoulli with $Pr[X=1]=1/2$. Then $I(X,Y)=0$. 2) Let $A=B=1$ (constants). Then $I(A,B)=0$. 3) Let $(W,Z) = \left\{\begin{array}{ll} (X,Y) & \mbox{with prob 1/2} \\ (A,B) & \mbox{with prob 1/2} ...


1

If for all $i$ we have $p_i<2^{-t}$ then $\log_2\frac{1}{p_i}> t$ for all $i$ hence \begin{equation} H(X) = \sum_{i=1}^{n}p_i\log_2\frac{1}{p_i}> t\sum_{i=1}^{n}p_i = t. \end{equation} This implies contradiction.


1

Suppose that for all $1\le i\le n$ you have $p_i < \frac{1}{2^t}$. $-H(x) = \sum\limits_{i=1}^{n}p_i\log_2p_i < \sum\limits_{i=1}^{n}p_i\log_2\frac{1}{2^t} = \log_2\frac{1}{2^t}=-t$, hence $H(x)>t$ contradiction. The first inequality follows from the simple fact that logarithm is increasing. note that your use of Jensens inequality is wrong. Take ...


1

Yes, the differential entropy disregards resolution (quantization). A continuous random variable can't be represented exactly with finite number of bits. By introducing an approximate representation through quantization you can relate to classical entropy. For example if you quantize uniformly with intervals of lenght $\Delta=2^{-n}$ you get a discrete ...


2

The differential entropy $h(x)$ is not a true generalization of the (discrete, true) entropy $H(X)$, only some of the properties of the later apply to the former. In particular, the property that $H(X)\ge 0$ , with $H(X)=0$ meaning "zero uncertainty" (or full knowledge), does not apply to $h(x)$. The differential entropy can be negative, and $h(x)=0$ has no ...


0

The correct answer is the first one, $H(X)=\log_2(3)=1.585\cdots$ The number you get by trying a binary choice decision tree corresponds, in essence to a coding of the form A : 0 B : 10 C : 11 which gives you a mean code length of $ 1 \times 1/3 + 2 \times 1/3 +2\times 1/3 =5/3 = 1.666\cdots$. This is indeed the optimum coding (or, equivalently, the ...


3

Differential Entropy can actually be negative, which is one of it's drawbacks. It just so happens that on $[0,1]$ all continuous distribution entropies are negative except for the uniform distribution. Let $h(x)$ be any continuous distribution on $[0,1]$ and $u(x)=1$ be the uniform distribution. Here's the proof using KL divergence notation:: $$0\leq ...



Top 50 recent answers are included