Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

What is an intuitive interpretation of the concept of maximum entropy? I want to understand this concept better but what I'm finding is too "advanced" right now. Can anyone simplify it ... imagine I'm 10 years old:) For starters i'm confused about what it's measuring, how it does that (roughly), and where one would apply it. Why do we talk of max entropy related to a Gaussian distribution vs, say, related to a uniform distribution. Are there different uses for this term (physics vs statistics?)

Thank you

share|improve this question

2 Answers 2

up vote 2 down vote accepted

Before knowing what is maximum entropy, the term entropy ($ -\sum \pi * log(\pi)$) should be very clear in your mind. Let me explain the entropy via an example directly. Take a coin and flip it. You have two possibilities, whether to obtain "head" or "tail". Now, imagine you flipped this coins 100 times and at the end you count how many times you got "head" and how many times you got "tail". There is one of the following possibilities:

  • You obtain only "head". Then P(head) = 1 and P(tail) = 0. The entropy here is 0.
  • You obtain only "tail". Then P(tail) = 1 and P(head) = 0. The entropy here is 0.
  • You obtain 50% "head" and 50% "tail". Then P(head) = 0.5 and P(tail) = 0.5. The entropy here is log(2).
  • for all other case, the entropy is less than log(2).

If you compare the 4 points above, you see that when you are "certain" abou the result (you get only head or you get only tail), your entropy is 0, which means there is not place for uncertainety. However, when you get log(2), this is because you have only two possibile states (tail or head) and they share the same probability and you are not sure at all which one to get. By this (log(2)), you have a maximum of certainty, which means the entropy is maximum.

Now, Maximum Entropy really is? It is a statistical inference method to construct a probability distribution with what we are given as information about the system. Information about the system are the constraints. The mandatory constraints is that the sum of probabilities should be equal to 1. You can add other constraints such as expected value (mean) or fluctuations (variance). - If you constraint is only sum of probabilities = 1, then you get a uniform distribution (like when tail and head have equal probability). - If you add a mean, you get an exponential distribution. - If you add mean and variance then you get a Gaussian distribution.

This method is inspired from physics and more precisely "thermodynamics". That is why we say it's about statistical physics. However, this method does not at all explain or (model) physical problem, but only statistical problems.

The subject is however far to be simple. In physics, we usually talk about Maximum Entropy Production and Minimum Entropy Production which represent the real "pysical" entropy production by the physical/biological/chemical systems and not the statistical data (although, there is an analogy, but the point should be crystal clear).

Hope this help ;)

share|improve this answer

Typically, we have some constraints on a distribution (for example, the marginal distributions), and we seek a max-entropy distribution satisfying these constraints. This basically means that, apart from the known constraints, we know as little as possible about the distribution. We do not want to specify anything about the distribution that is not already implied by the known constraints.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.