# Tag Info

## Hot answers tagged information-theory

3

Let $n=|A|+|B|$. Use $\lceil \log_2 (n+1)\rceil$ bits to binary encode $|A|$ (a number between $0$ and $n$, inclusive), then copy $A$, then copy $B$. To decode $C$, start with $n=|C|$ and while $\lceil \log_2(n+1)\rceil + n>|C|$ decrease $n$. Then interprete the first $\lceil \log_2 (n+1)\rceil$ as length of $A$, voila. Efficiency: There are ...

3

i have have calculated kullback-leibler divergence which is equal 0.492820258 The KL divergence is not a dimensionless number, it has an unit (which depends on the base of the logarithm used), you must specify it unless it's implied by the context. I guess you used base 2, hence the unit it bits. The KL divergence (or "distance") is not symmetric: $D(p ... 3 You wanted an intuitive interpretation of entropy$H(X)$entropy. Let me share with you one way that I understand entropy. Note that my explanation will be mostly intuitive rather than rigorously mathematical. Let me start by giving you my interpretation first and then let me justify it. Entropy can be viewed as the cost of encoding a specific ... 2 The capacity of channels with deletions and insertions remains poorly understood. Still some lower and upper bounds are known, you can easily google for that. As for codes that work under such circumstances you also have some literature to browse, I'd suggest that you take a look at the seminal work by MacKay and Davey: "Reliable communication over ... 1 Lines$(3)$and$(4)$provide the definition for the notation in line$(2)$for$P_{dist}()$and$P()$. The only function/notation left undefined in these lines is$C(w_i)$in line$(3)$, but I'm pretty sure this is just the count of$w_i$in the documents (i.e. the number of times$w_i$occurs). To explain what I think is going on here, I'll start by ... 1 Just some hints. We can consider this as an additive gaussian channel, with noise$Z$, bounded input$X$and output$W=X+Z$. Furthermore,$h(W) = h(Z) + h(X) - h(X|W)=h(Z)+I(X;W)$For the first equality see here. Because$h(Z)$is fixed, our problem of maximizing$h(W)$is then equivalent to finding the pdf for$X$that maximizes the mutual information, ... 1 You are right, you don't need to bother with the "multivariate mutual information" you link -that's a rather esoteric and dubious extension of the proper mutual information. You need nothing special, just the usual definition of$I(Y;X)$. That$X=(X_0,X_1)$is a multivariate variable does not really change anything:$$I(Y;X_0,X_1) = \sum_{X_0,X_1,Y} ... 1 Perhaps we need more context. By itself, and using the common definition of information, the statement is false. For example, take:$P(A)=1/2$, and$P(B)=1$. The amount of information of event$B$is zero. It makes no sense to say that$A$gives usa twice information than that. 1 "whether it is true that$H(X∣Y1,Y2)=H((X∣Y1)∣Y2)$" Basically yes, but the second notation is not very correct, it means nothing. You can't write$(X|Y)$as it were a random variable (don't confuse it with$(X|Y=y)\$, alternative notation used in probability, which is indeed a new random variable - but that's not what we mean in the conditioned entropy ...

1

If the log-likelihood is quadratic (i.e., the estimator is normally distributed), then the Fisher information is the reciprocal of the variance of the estimator (hence, the lower the variance of the estimator, the more "information" is provided by the data about the parameter...loosely speaking). The square root of the reciprocal of the Fisher Information is ...

Only top voted, non community-wiki answers of a minimum length are eligible