1
$\begingroup$

I am trying to understand maximum entropy modelling and I came across log likelihood equation of the empirical distribution, which I did not quite understand, which also eventually turns out to be equal to the dual function we get when trying to maximise the entropy with constraints using Lagrange's multipliers

$$ L_\widetilde{p}(p) \equiv log \prod\limits_{x,y}p(y|x)^{\widetilde{p}(x,y)} = \sum\limits_{x,y}\widetilde{p}(x,y)\log{p(y|x)}$$

Where,

  • $y$ is the outcome produced by a random process
  • $x$ is the feature influencing the outcome $y$
  • $L_\widetilde{p}(p)$ is the log likelihood
  • $\widetilde{p}$ is the empirical distribution of training data
  • $p(y|x)$ is the model
  • $\widetilde{p}(x,y) \equiv \frac{1}{N} \times \text{number of times that } (x,y) \text{ occurs in the sample} $

Can someone please explain how $p(y|x)$ is raised to the power of $\widetilde{p}(x,y)$ in the log likelihood equation mentioned above? Instead, shouldn't $p(x|y)$ be raised to the power of number of times that $(x,y)$ occurs in the sample.

I went through this reference tutorial on max entropy.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.