0
$\begingroup$

I am trying to figure out how the equation for factorized entropy below is derived. The equation for entropy is $H(Q) = -\sum_x Q(x)\log Q(x)$ where $Q$ is a probability distribution. Let $Q(x) = \prod_i Q_i(x_i)$ (a factorized distribution). Then, this is the derivation(see page 4, slide 1) I have seen for factorized entropy:

$$ H(Q)= - \sum_x Q(x)\log Q(x)$$ $$=-\sum_x(\prod_{i\in x} Q(x_i))\log(\prod_{x_i \in x} Q(x_i))$$ $$=-\sum_x(\prod_{i} Q(x_i))\sum_i \log Q(x_i))$$ $$=-\sum_i\sum_{x_i}Q(x_i)\log Q(x_i)$$

I can't figure out how the last line was derived(replacement of $\prod_iQ(x_i)$ by $\sum_{x_i}Q(x_i)$) . What confuses me is that $ab (\log a + \log b) \equiv \log (ab)^{ab}$ is not same as $(a+b)(\log a + \log b) \equiv \log (ab)^{a+b}$. I will appreciate pointers on how the last line is derived.

$\endgroup$

1 Answer 1

0
$\begingroup$

$\text{Line 3 }=-\sum_x(\Pi_jQ_j(x_j))\sum_i\log Q_i(x_i)=-\sum_xQ(x)\sum_i\log Q_i(x_i)=-\sum_i\sum_xQ(x)\log Q_i(x_i)=-\sum_i\sum_{x_i,x_{-i}}Q_{-i}(x_{-i})Q_i(x_i)\log Q_i(x_i)=-\sum_i(\sum_{x_{-i}}Q_{-i}(x_{-i}))(\sum_{x_i}Q_i(x_i)\log Q_i(x_i))=-\sum_i\sum_{x_i}Q_i(x_i)\log Q_i(x_i)=\text{last line}$

$x_{-i}$ means non-$x_i$ terms.

$\sum_{x_i,x_j,x_k}Q_i(x_i)Q_j(x_j)Q_k(x_k)=\sum_{x_i}Q_i(x_i)\sum_{x_j}Q_j(x_j)\sum_{x_k}Q_k(x_k)$, as long as $x_i,x_j,x_k$ are pairwise/mutually independent, which is necessary for $Q$ to be factorized.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.