I'm not sure how to pose the question, so I'll just try to explain. It's inspired by reading lecture notes on statistical mechanics (section 2 on page 3).

Let's say I have an unknown probability distribution $\{p_i\}_{i=1}^N, \sum_i^N p_i = 1, 0 \le p_i \le 1$ (for simplicity assuming finite number of possible outcomes). All I know are some averages obtained with these $p_i$s, e.g. $$ \sum_{i=1}^N p_i x_i = X $$

$$ \sum_{i=1}^N p_i y_i = Y $$

etc. All $x_i$, $y_i$, $X$, $Y$, ... are a given, but the number of such equations is less than $N$, so it's not possible to uniquely determine all $p_i$s. What I want to find is "the best" solution for $\{p_i\}_{i=1}^N$. Let's say that it should deviate from uniform distribution ($p_i = 1/N$ for all $i$) "as little as possible". Is there any rigor concept for such a task in mathematics?

E.g. in statistical mechanics, one should find a set of $p_i$s that maximizes entropy $$ S = -\sum_{i=1}^N p_i \ln p_i $$ given the constraints (this way you can obtain Gibbs distribution if average energy $U$ of the system is a given: $\sum_i p_i = 1, \sum_i p_i E_i = U \implies p_i = \exp(-E_i/T)/Z$).

On the other hand, without reading the lecture notes above and knowing nothing about the entropy, I'd just try to minimize $\chi^2$ instead: $$ \chi^2 = \sum_{i=1}^N \left( p_i - 1/N \right)^2 $$ given the same constraints, this way I would get a different set of $p_i$

Obviously, there are many more ways to do it.

But is there any rigor concept that I could use to compare the results? Maybe a way to define how "far" one distribution is from another, so that I could say that the one obtained by maximizing entropy is "closer" to a uniform distribution than the one obtained by minimizing $\chi^2$?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.