What is an intuitive interpretation of the concept of maximum entropy?
I'm confused about what it measures, how it does that (roughly), and where one would apply it. Why do we talk of maximum entropy related to a Gaussian distribution vs, say, related to a uniform distribution? Are there different uses for this term (physics vs statistics)?
Before knowing what is maximum entropy, the term entropy ($ -\sum \pi * log(\pi)$) should be very clear in your mind. Let me explain the entropy via an example directly. Take a coin and flip it. You have two possibilities, whether to obtain "head" or "tail". Now, imagine you flipped this coins 100 times and at the end you count how many times you got "head" and how many times you got "tail". There is one of the following possibilities:
If you compare the 4 points above, you see that when you are "certain" abou the result (you get only head or you get only tail), your entropy is 0, which means there is not place for uncertainety. However, when you get log(2), this is because you have only two possibile states (tail or head) and they share the same probability and you are not sure at all which one to get. By this (log(2)), you have a maximum of certainty, which means the entropy is maximum.
Now, Maximum Entropy really is? It is a statistical inference method to construct a probability distribution with what we are given as information about the system. Information about the system are the constraints. The mandatory constraints is that the sum of probabilities should be equal to 1. You can add other constraints such as expected value (mean) or fluctuations (variance). - If you constraint is only sum of probabilities = 1, then you get a uniform distribution (like when tail and head have equal probability). - If you add a mean, you get an exponential distribution. - If you add mean and variance then you get a Gaussian distribution.
This method is inspired from physics and more precisely "thermodynamics". That is why we say it's about statistical physics. However, this method does not at all explain or (model) physical problem, but only statistical problems.
The subject is however far to be simple. In physics, we usually talk about Maximum Entropy Production and Minimum Entropy Production which represent the real "pysical" entropy production by the physical/biological/chemical systems and not the statistical data (although, there is an analogy, but the point should be crystal clear).
Hope this help ;)
Typically, we have some constraints on a distribution (for example, the marginal distributions), and we seek a max-entropy distribution satisfying these constraints. This basically means that, apart from the known constraints, we know as little as possible about the distribution. We do not want to specify anything about the distribution that is not already implied by the known constraints.
Maximum entropy is a guiding principle in assigning probabilities to events. When nothing is known about the problem you can assign the same probability to all likely states resulting in a uniform probability distribution. for example 1/2 to both events (head, tail) in a coin tossing experiment or say 1/6 to each of the faces in a die tossing experiment.
However, what if you know something about the problem e.g in the die tossing experiment the average value was 4.5 after n trials? You can of course ignore such information and still assign probabilities as before. But that is not taking advantage of the available information. In such cases you can apply the principle of Maximum entropy. The probabilities will then be
a) in accordance with information that is known. and, b) no hidden biases will be introduced.
In other words you don't want to make unjustified assumptions while taking advantage of what you already know in assigning probabilties.
This paper by Jaynes is a great resource for anyone trying to understand what the maximum entropy principle is and how to use it.
