3
$\begingroup$

I hope this isn't any kind of duplicate. I have looked through the forum, but it's possible I overlooked something.

I'm currently in the 4th semester of my physics studies and listened to some additional mathematics lectures. I stumpled across the terms "information theory", "information field theory" and "information geometry". I have absolutely no background in these topics, however, I think they sound really interesting after I looked them up.

For me it's really hard to understand the connections between these three similar(?) branches of mathematics/physics. I haven't found much on the internet either (no direct comparisons etc.), so I guess this discussion will be very informative for a lot of interested people new to these fields.

It would be great if people who are somehow familiar with these topics could explain differences and similarities, their applications in physics and other branches and maybe about their "future potential".

I would love to hear as much as possible from you! :)

$\endgroup$
2
$\begingroup$

Information theory is (unsurprisingly) the study of quantifying information, usually through the concept of entropy, the expected information content of a random variable, and various metrics on the space of probability distributions, such as the KL-divergence. The origin of information theory is somewhat computer sciency, relying on probability theory to do things like determine the optimal amount data can be compressed (obviously related to the information content of the data) or information transmission (i.e. sending information over noisy channels).

Two nice books are Elements of information theory by Cover and Thomas, and the book that is often considered to have founded the field, The Mathematical Theory of Communication by Shannon. For connections to physics, check out Information Theory and Statistical Mechanics by Jaynes as well.

The mathematical tools of information theory, though, are quite physicsy. Most obvious is the relation between the Boltzmann equation (or Gibb's entropy) $S$ with microstate probabilities $p_i$, and the information entropy $\mathbb{H}$ of a discrete random variable $X$ with probability mass values $p(x_i)$: $$ S = -k_B\sum_i p_i \ln p_i\;\;\;\&\;\;\; \mathbb{H}[X]= -\sum_i p(x_i) \ln p(x_i) $$ See also this question and this question. Another interesting connection can be found via the Boltzmann distribution: $$ p_i = \frac{\exp(-E_i/c)}{\sum_\ell \exp(-E_\ell/c)} $$ where $c=k_BT$ and $E_j$ is the energy. (Others may know this as the softmax function). In machine learning, one often wishes to construct a function that reproduces a mapping between two vector spaces. One way to regularize the learner is to ensure that the information held by the parameters of the learned mapping is less than that of the data they are learning from, since such "extra information" is extraneous. So we wish to minimize the amount of information required to communicate the model parameters. This code cost can be written as a Helmholtz Free Energy equation. However, it turns out that that the Helmholtz free energy equation is minimized by the the Boltzmann distribution! Hence, the Boltzmann distribution from statistical physics also has close relations to information theory. Of course, this was a rather inexact description; see Hinton and van Camp, Keeping Neural Networks Simple by Minimizing the Description Length of the Weights for details.

In general, information theory has penetrated various parts of physics, as a different outlook on the same phenomena. Maybe see Information, Physics, and Computation by Mezard et al. Perhaps one of the most common connections in the mainstream press is through black holes holding information on their surface. See Bekenstein, Black Holes and Information Theory. There's also connections through quantum information theory, e.g. through the von Neumann entropy, which appears in analyses related to quantum computing, but I don't know much about that.

Overall, I'd say that information theory is a very fundamental field, which to me studies essentially studies properties of probability distributions. Hence anywhere probability and statistics appears, so too could information theory.


Information Field Theory is about using Bayesian inference to handle measurement uncertainty and finiteness, specifically for physical problems. This lets you combine prior information (e.g. from theoretical physics) with data (i.e. from measurements). So obviously there's a connection to applied physics, in that the theory is designed to solve problems in that field. However, there are some clear connections between physics and information theory within the mathematics of the framework itself.

Briefly, given data $d$ for a field $s$, Bayes Theorem says we can infer the field value based on the predetermined prior $P(s)$ and the data likelihood $P(d|s)$: $$ P(s|d) = P(d|s)P(s)/P(d) = P(d,s)/P(d) = \exp(-H[d,s])/P(d) $$ where $H[d,s]=H[d|s] + H[s] = -(\ln P(d|s) + \ln P(s))$ is the "information Hamiltonian". For instance, one could choose the prior distribution to be Gaussian, and also (by assuming Gaussian noise in the measurements) the likelihood to be Gaussian as well. Clearly, under this Bayesian model, the most likely field is given by minimizing the information Hamiltonian. Some other connections between KL divergence and Shannon entropy, "information energy", and Gibbs Free energy can also be seen here.


Information geometry is a geometric perspective on probability distributions. Essentially, consider a parametric probability distribution. The space of distributions covered by varying these parameters forms a Riemannian manifold (a statistical manifold), with a metric tensor given by the Fisher information matrix. Thus, we can do things like have properly defined geodesic distances between probability distributions.

If you're familiar with general relativity, then you know that Riemannian manifold theory underlies both information geometry and spacetime in relativistic physics. I don't know much about connections to physics here, but you might like to look at Caticha and Cafaro, From Information Geometry to Newtonian Dynamics.

See also this question for more information and resources. To my knowledge, information field theory is relatively separate from information geometry, but both rely on information theory.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ The Mathematical Theory of Communication is not a book; it is a paper :D (or, I suppose, an article) (though I can definitely see why you might want to call it a book) $\endgroup$ – E-A Jun 8 '18 at 22:45
  • 1
    $\begingroup$ @E-A I suppose you're right, but I own the book form here, so it first comes to my mind as a (short) book. XD $\endgroup$ – user3658307 Jun 9 '18 at 0:13
  • $\begingroup$ Whoa; didn't know they republished it as a book! Thanks for informing me :D $\endgroup$ – E-A Jun 9 '18 at 1:34
  • $\begingroup$ Thanks a lot for this elaborate comment! I will leave this question unanswered for bit longer, so maybe more people can contribute. $\endgroup$ – Marc Jun 10 '18 at 0:59
  • $\begingroup$ @Marc Hopefully they do! $\endgroup$ – user3658307 Jun 10 '18 at 1:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.