We have a discrete probability distribution $P(X)$ and $\Omega = \{X_i\}$ is the sample space. (E.g. if the experiment is tossing a fair coin, $ \Omega =\{Head, Tail\}$ and $\forall X_i\ P(X_i)=0.5$)

Then, how can you interpret the sum of log-likelihood: $\sum_{X_i \in \Omega}\log P(X_i)$ ? Is this quantity a basic one in the probability theory or the information theory like Entropy? (Of course, $\sum_{X_i \in \Omega} P(X_i) = 1$, though.)

I think this quantity is closely relating to Shannon Entropy because it is the likelihood of occurring all event:

$$ \sum_{X_i \in \Omega}\log P(X_i) = \log \prod_{X_i \in \Omega} P(X_i) $$

because, I guess that $\prod_{X_i \in \Omega} P(X_i) \propto S(X)$ holds. Here, S(X) is Shannon Entropy i.e. $S(X) = -\sum_{X_i}P(X_i) \log P(X_i)$.

Update 1 on 2019/08/23

I apologize that I have used the term "likelihood" without a clear definition.

I have defined implicitly "the likelihood" as the probability of the trial of $|\Omega|$ times under i.i.d. assumption as follow: $$ \text{Likelihood} = \prod_{k=1}^{|\Omega|} P(X_k), $$ where $\{X_k\}_{k=1,...,|\Omega|}$ is a sequence of samples drawn from $P(X)$.

Update 2 on 2019/08/23

After posting the question, I found that the quantity is employed as a function that characterizing Itakura-Saito divergence. With the notations on the Wikipedia page of Bregman divergence, If $F(p) = -\sum_{X_i \in \Omega}\log P(X_i)$, Bregman divergence becomes Itakura-Saito divergence.

  • $\begingroup$ "the likelihood of occurring all event" I have no idea what that means. And what is $S(X_i)$ ? (and how can it depend on $i$ where the LHS does not?) $\endgroup$ – leonbloy Aug 22 '19 at 17:06
  • $\begingroup$ @leonbloy Thank you for suggesting my mistake. I updated my question. $\endgroup$ – rkjt50r983 Aug 23 '19 at 0:07

No, I don't think that quantity is a "basic one in the probability theory or the information theory". And I very much doubt it has some useful meaning. For one thing, it's rather ill behaved in the sense that it tends to minus infinity if one of the probabilities tends to zero (hence a sample space $\{0,1,2\}$ with $P(0)=P(1)$ and $P(2)=\epsilon \to 0$ gives a totally different result than a variable equiprobable over the sample space $\{0,1\}$ ... which is unsatisfactory).

And it's not the "sum of log-likelihood" , nor is $\prod_{X_i \in \Omega} P(X_i)$ a likelihood (at most, it's the likelihood of a very particular realization).

  • $\begingroup$ I updated the question in order to relate the quantity with likelihood. I understand your suggestion that the quantity is unsatisfactory. However, I found that this quantity is employed at least in the context of filtering theory as the characteristic function of Itakura-Saito divergence as mentioned in the second updated comment in my question. $\endgroup$ – rkjt50r983 Aug 23 '19 at 0:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.