I am trying to understand maximum entropy modelling and I came across log likelihood equation of the empirical distribution, which I did not quite understand, which also eventually turns out to be equal to the dual function we get when trying to maximise the entropy with constraints using Lagrange's multipliers

$$ L_\widetilde{p}(p) \equiv log \prod\limits_{x,y}p(y|x)^{\widetilde{p}(x,y)} = \sum\limits_{x,y}\widetilde{p}(x,y)\log{p(y|x)}$$


  • $y$ is the outcome produced by a random process
  • $x$ is the feature influencing the outcome $y$
  • $L_\widetilde{p}(p)$ is the log likelihood
  • $\widetilde{p}$ is the empirical distribution of training data
  • $p(y|x)$ is the model
  • $\widetilde{p}(x,y) \equiv \frac{1}{N} \times \text{number of times that } (x,y) \text{ occurs in the sample} $

Can someone please explain how $p(y|x)$ is raised to the power of $\widetilde{p}(x,y)$ in the log likelihood equation mentioned above? Instead, shouldn't $p(x|y)$ be raised to the power of number of times that $(x,y)$ occurs in the sample.

I went through this reference tutorial on max entropy.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.