Given two random variables $X, Y$, what is the joint distribution $P(X,Y)$ that maximizes entropy $H(X,Y)$, subject to given marginal probabilities $P(X), P(Y)$ and given values along the diagonal of the probability matrix (i.e, $P(X=i,Y=i) = d_i$)?

A little background: I'm a PhD student writing a course final paper exploring measures on a finite-domain, discrete-time, definitely-not-Markovian process. (In particular, it's the app that is open on someone's phone at a moment in time.) I started by focusing on mutual information between moments at varying time separations - much like autocorrelation but instead of correlation it's mutual information. (By the way, if someone knows a name for or has a paper reference to either that or anything slightly related, let me know - I'm enjoying the mathematics a lot, but I'm sorely underqualified to do this kind of stuff.)

I'd like to be able to break down each slice of the autocorrelation - each moment of 'predictability' (mutual information) - into information gained from when the samples have the same app (e.g. you're still browsing Facebook ten minutes later) and when app usage 'clusters' (e.g, a mobile game means you're at home relaxing which means Facebook is more likely than Outlook). My intuition is to use the principle of maximum entropy on the joint distribution subject to the important constraints (marginals and 'diagonal' probabilites) to find this amount. Hence the question above!


My solution was a rather inelegant one. The proof sketch follows the idea that maximizing entropy among the non-diagonal items implies there exists some vectors $v_x$ and $v_y$ whose outer product produces all non-diagonal entries (note the diagonal entries are discarded).

I then used this fact to estimate $v_x$ and $v_y$ with gradient descent. Initial values were the marginal probabilites minus the diagonals, and then in each step the joint probability matrix is estimated (outer product, but replace the diagonal with the given diagonal values) then the error is computed upon the marginal probabilites.

There is a more elegant strategy I'm sure, but I'm happy with this.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.