Tag Info

New answers tagged

0

The right hand side can be proved with the joint convexity of relative entropy, that is, H(\sum_{i}c_i P_i||\sum_i c_iQ_i)\leq \sum_i c_i H(P_i||Q_i) the left hand side is not valid as I see. For example, suppose P_1=Q_2=(0.3,0.7),P_2=Q_1={0.7,0.3}, and a=0.5, then the middle term vanishes while the left side is positive.


0

The possible values of the triplet $A,B,C$ are $\{(1,1,1),(1,1,-1),\cdots,(-1,-1,-1)\}.$ Based on a sample the probabilities of the the different $8$ outcomes could be estimated. Let those probabilities be denoted by $p_{1,1,1},p_{1,1,-1},\cdots,p_{-1,-1,-1}$. The entropy of $(A,B,C)$, by definition, is $$H(A,B,C)=-(p_{1,1,1}\log ...


0

Beyond the symmetric KL-divergence, Information Theoretic Learning presented several symmetric distribution "distances". The idea is just to realize that pdfs are like any other functions in a L2-space. Thus, you can calculate the Euclidian distance $\int_x(p(x)-q(x))^2dx$, Cauchy-Schwarz distance, etc. There are even approximations to these distances ...


2

You seem to have two output classes and given the hint, the inputs must be equally likely since 2 of 10 inputs lead to class 1, the rest class zero, corresponding to 0.2 and 0.8 respectively. The definition of entropy is $$-\sum_{x} p_x \log_2(p_x)$$ which will give the numerical value of $$-0.2 \times (-2.3) -0.8 \times (-0.3)$$as the entropy.


0

The post you are referring to is about differential entropy $h(X)$. It's true that simply relabeling the outcomes of a discrete random variable $X$ cannot change its entropy $H(X) = -\sum_x p(x)\log p(x)$. Differential entropy is a similar quantity, but lacks some of the nice properties of $H(X)$, e.g., it can be negative, it changes with nonzero scaling, ...


0

Ok, this is not a definite answer, but just some pointers to try and help build some intuition. Entropy is an expected value, specifically this one: $-E[log_n(p)]$. Then recall that logarithm measures the number of digits of the representation of a number. Log 10 of a base 10 number is a measure of how far from the decimal point it is $10^3 = 1000$ which ...


0

If the probabilities are $p_i = 2^{-k}$ ("diadic distribution"), then it's easy (assuming you understand how Hufmman codes are constructed) to show that the binary Huffman code will give a perfectly balanced binary tree, of depth $k$. Hence, all codewords will have length $k=\log (1/p_i)$


1

If there are $n$ unique items in the set $S$ each with equal (conditional) probability of occurrence then: $$\begin{align} \mathsf H(S) & = - \sum_{i=1}^n \tfrac 1 n \log_2(\tfrac 1 n) \\[2ex] & = \log_2 n \end{align}$$


0

Since the peak of the skewed tent touches ocurrs at $(\alpha, 1)$, the pre-image of this point will consist of two points, therefore its second iterate will have four laps. But now we can apply this argument to both these new points (only because of the surjectivity of the function ensured by the peak of the tent "touching the top"). In general the critical ...


0

This is not a complete answer, I am writing here because SE does not like long lines of comments. If you consider the dynamics on the space of admissible sequences of configurations i.e. on the space of all possible (one-sided) trajectories of your $\mathfrak{C}+D$ map on the space of configurations, then it is indeed corresponds to the shift on a certain ...


2

Your calculation is wrong. The correct answer for four digits is $$1.7820 .$$ The correct interpretation of the entropy is as follows. Assume that you have to code an infinite sequence of independent random variables distributed identically; taking different values (symbols) with the given probabilities. Then create disjoint sequence of length $N$ out of ...


0

This is basically Fano's inequality. Sketch: Let $x$ be a random variable such that $x=1$ iff $a=b$, $x=0$ otherwise. Then use the chain rule (conditioned on $b$): $$H(a,x|b)=H(a|x,b) + H(x|b) = H(x|a,b)+H(a|b)$$ Then note that $H(x|a,b)=0$, and $H(x|b)\le H(x)\le 1$. Furthermore, $H(a|x,b)=P(x=0) H(a|x=0,b) + P(x=1)H(a|x=1,b)$ with $H(a|x=1,b)=0$ and ...


0

$I(a;b) = \frac{3h}{4}$ and $I(a;c) = \frac{3h}{4}$. Hence, $I(a;b;c) \ge \frac{h}{2}$. So if $I(b;c) \ne I(a;b;c)$ then $I(b;c) > \frac{h}{2}$ else $I(b;c) \ge \frac{h}{2}$.



Top 50 recent answers are included