15
$\begingroup$

When I first got into information theory, information was measured or based on shannon entropy or in other words, most books I read before were talked about shannon entropy. Today someone told me there is another information called fisher information. I got confused a lot. I tried to google them. Here are links, fisher information: https://en.wikipedia.org/wiki/Fisher_information and shannon entropy goes here https://en.wikipedia.org/wiki/Entropy_(information_theory).

What are differences and relationship between shannon entropy and fisher information? Why do two kinds of information exist there?

Currently, my idea is that it seems fisher information is a statistical view while shannon entropy goes probability view.

Any comments or anwsers are welcome. Thanks.

$\endgroup$
8
$\begingroup$

Fisher information is related to the asymptotic variability of a maximum likelihood estimator. The idea being that higher Fisher Information is associated with lower estimation error.

Shannon Information is totally different, and refers to the content of the message or distribution, not its variability. Higher entropy distributions are assumed to convey less information because they can be transmitted with fewer bits.

However, there is a relationship between Fisher Information and Relative Entropy/KL Divergence, as discussed on Wikipedia.

$\endgroup$
  • $\begingroup$ What is the definition of asymptotic variability? $\endgroup$ – Bear and bunny Nov 11 '15 at 13:48
  • 2
    $\begingroup$ @Bearandbunny basically, the fisher information can be interpreted as the inverse of the standard error (squared), but only when the log-likelihood is quadratic (i.e., the Gaussian log-likelihood). In the vast majority of cases, we are not dealing with a gaussian population..but, the log-likelihood of the MLE will often rapidly converge to a quadratic, especially around $\pm2$ standard deviations. In this case, treating the inverse of the fisher information as an estimated precision of an estimate will be approximately correct. $\endgroup$ – user237392 Nov 11 '15 at 14:05
  • 2
    $\begingroup$ @Bearandbunny here is a paper that outlines the theory: sites.stat.psu.edu/~sesa/stat504/Lecture/lec3_4up.pdf $\endgroup$ – user237392 Nov 11 '15 at 14:07
  • $\begingroup$ I see. will upvote. $\endgroup$ – Bear and bunny Nov 11 '15 at 14:17
  • $\begingroup$ It is not true that "higher entropy distributions ... can be transmitted in fewer bits". For instance Bernuolli distribution with $p=1/2$ has entropy of 1 bit while Bernuolli with $p=1/10$ of about 1/3 bit and this is exactly because it conveys less information. $\endgroup$ – sztal Apr 23 at 15:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.