Let $p(x,y,z)$ be a probability distribution over 3 variables (suppose them discrete, but it shouldn't matter). I know that the distribution with maximal entropy which preserves the first order marginals $p(x),p(y),p(z)$ is simply the product $p(x)p(y)p(z)$.

What is instead the distribution with maximal entropy which preserves these marginals: $p(x,y),p(x,z),p(y,z)$? Or if there is no closed form, can I define it implicitly?


  • $\begingroup$ My intuition tells me you will only be able to preserve one of $p(x,y)$, $p(x,z)$ or $p(y,z)$ at a time unless you take the full joint probability. $\endgroup$ – Dan Mar 11 '15 at 13:48
  • $\begingroup$ Why? Just take a lower entropy. $\endgroup$ – geodude Mar 11 '15 at 13:52
  • $\begingroup$ I'm not sure I follow, could you give an example? $\endgroup$ – Dan Mar 11 '15 at 13:56
  • $\begingroup$ There always exist a distribution which has all the same double marginals, namely the original $p$. And possibly some other...among which we take the one with the highest entropy. If it were impossible to find others, it would mean that the double marginals determine $p$ uniquely, which is absurd. $\endgroup$ – geodude Mar 11 '15 at 14:02
  • $\begingroup$ Why is it absurd that all three double marginals taken together uniquely define $p$? I'm trying to find an example where that is not the case but so far I've been unsuccessful. $\endgroup$ – Dan Mar 11 '15 at 14:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.