# What are differences and relationship between shannon entropy and fisher information?

When I first got into information theory, information was measured or based on shannon entropy or in other words, most books I read before were talked about shannon entropy. Today someone told me there is another information called fisher information. I got confused a lot. I tried to google them. Here are links, fisher information: https://en.wikipedia.org/wiki/Fisher_information and shannon entropy goes here https://en.wikipedia.org/wiki/Entropy_(information_theory).

What are differences and relationship between shannon entropy and fisher information? Why do two kinds of information exist there?

Currently, my idea is that it seems fisher information is a statistical view while shannon entropy goes probability view.

Any comments or anwsers are welcome. Thanks.

• @Bearandbunny basically, the fisher information can be interpreted as the inverse of the standard error (squared), but only when the log-likelihood is quadratic (i.e., the Gaussian log-likelihood). In the vast majority of cases, we are not dealing with a gaussian population..but, the log-likelihood of the MLE will often rapidly converge to a quadratic, especially around $\pm2$ standard deviations. In this case, treating the inverse of the fisher information as an estimated precision of an estimate will be approximately correct. – user237392 Nov 11 '15 at 14:05
• It is not true that "higher entropy distributions ... can be transmitted in fewer bits". For instance Bernuolli distribution with $p=1/2$ has entropy of 1 bit while Bernuolli with $p=1/10$ of about 1/3 bit and this is exactly because it conveys less information. – sztal Apr 23 at 15:44