# Why is $\int q(\theta) \ln q(\theta) d\theta =\sum_i \int q_i(\theta_i) \ln q_i(\theta_i)d\theta_i$ if $q(\theta)=\prod_i q_i(\theta_i)$?

The context of this question is variational inference (vriational Bayes) assuming factoring posterior distributions, which is then also known as mean field theory. However the argument is purely algebraically.

Let $q(\theta)$ be a function where $\theta$ a parameter vector. Assume $$q(\theta)=\prod_{i=1}^D q_i(\theta_i),$$ where $\int q_i(\theta_i) d \theta_i=1$. Then another function is given by $$\int q(\theta) \ln q(\theta) d\theta$$ which, under the assumption made above, apparently factorizes to

$$\sum_{i=1}^D \int q_i(\theta_i) \ln q_i(\theta_i)d\theta_i.$$

I do not understand why. I can see that

$$\prod_{i=1}^D \int q_i(\theta_i) \sum_{j=1}^D \ln q_j(\theta_j) d\theta_i$$

but I do not know how to get to the result from here.

• You need to have different indices on the sum and the product or it's not clear what's going on. May 26, 2017 at 21:42
• @Chappers I tried to edit - better? May 26, 2017 at 21:55
• Yes, much. Do we know anything about $\int q_i(\theta_i)d\theta_i$? May 26, 2017 at 21:57
• @Chappers Yes, actually. I missed that: $\int q_i(\theta_i) d \theta_i= 1$ (it's a distribution over $\theta$). May 26, 2017 at 21:58

Let's expand the log first, so that the sum comes outside: $$\int q(\theta) \log{q(\theta)} \, d\theta = \int q(\theta) \sum_i \log{q_i(\theta_i)} \, d\theta = \sum_i \int q(\theta) \log{q_i(\theta_i)} \, d\theta.$$ Now expand $q(\theta)$, $$\int q(\theta) \log{q_i(\theta_i)} \, d\theta = \int \left(\prod_j q_j(\theta_j) \right) \log{q_i(\theta_i)} \, d\theta.$$ Separate off the $q_i(\theta_i)$ from the product, and divide the integral up: $$\int \left(\prod_j q_j(\theta_j) \right) \log{q_i(\theta_i)} \, d\theta = \int \left(\prod_{j\neq i} q_j(\theta_j) \right) q_i(\theta_i) \log{q_i(\theta_i)} \, d\theta \\ = \left( \int q_i(\theta_i) \log{q_i(\theta_i)} \, d\theta_i \right) \left( \prod_{j \neq i} \int q_j(\theta_j) \, d\theta_j \right).$$ The integrals in the second bracket are all $1$, and we get the answer.
It will probably suffice to see it worked out when $D = 2$:
Now the first integral is separable: \begin{align*} \iint q_1(\theta_1)q_2(\theta_2)\ln(q_1(\theta_1))\,d\theta_1\,d\theta_2&= \int q_2(\theta_2)\,d\theta_2\cdot\int q_1(\theta_1)\ln(q_1(\theta_1))\,d\theta_1. \end{align*} And likewise the second is also separable: \begin{align*} \iint q_1(\theta_1)q_2(\theta_2)\ln(q_2(\theta_2))\,d\theta_1\,d\theta_2&= \int q_1(\theta_1)\,d\theta_1\cdot\int q_2(\theta_2)\ln(q_2(\theta_2))\,d\theta_2. \end{align*} Since $\int q_i = 1$, you know how to take it from here.