0
$\begingroup$

The context of this question is variational inference (vriational Bayes) assuming factoring posterior distributions, which is then also known as mean field theory. However the argument is purely algebraically.

Let $q(\theta)$ be a function where $\theta$ a parameter vector. Assume $$q(\theta)=\prod_{i=1}^D q_i(\theta_i),$$ where $\int q_i(\theta_i) d \theta_i=1$. Then another function is given by $$\int q(\theta) \ln q(\theta) d\theta$$ which, under the assumption made above, apparently factorizes to

$$\sum_{i=1}^D \int q_i(\theta_i) \ln q_i(\theta_i)d\theta_i.$$

I do not understand why. I can see that

$$\prod_{i=1}^D \int q_i(\theta_i) \sum_{j=1}^D \ln q_j(\theta_j) d\theta_i$$

but I do not know how to get to the result from here.

$\endgroup$
4
  • $\begingroup$ You need to have different indices on the sum and the product or it's not clear what's going on. $\endgroup$
    – Chappers
    May 26, 2017 at 21:42
  • $\begingroup$ @Chappers I tried to edit - better? $\endgroup$
    – tomka
    May 26, 2017 at 21:55
  • $\begingroup$ Yes, much. Do we know anything about $\int q_i(\theta_i)d\theta_i$? $\endgroup$
    – Chappers
    May 26, 2017 at 21:57
  • $\begingroup$ @Chappers Yes, actually. I missed that: $\int q_i(\theta_i) d \theta_i= 1$ (it's a distribution over $\theta$). $\endgroup$
    – tomka
    May 26, 2017 at 21:58

2 Answers 2

2
$\begingroup$

Let's expand the log first, so that the sum comes outside: $$ \int q(\theta) \log{q(\theta)} \, d\theta = \int q(\theta) \sum_i \log{q_i(\theta_i)} \, d\theta = \sum_i \int q(\theta) \log{q_i(\theta_i)} \, d\theta. $$ Now expand $q(\theta)$, $$ \int q(\theta) \log{q_i(\theta_i)} \, d\theta = \int \left(\prod_j q_j(\theta_j) \right) \log{q_i(\theta_i)} \, d\theta. $$ Separate off the $q_i(\theta_i)$ from the product, and divide the integral up: $$ \int \left(\prod_j q_j(\theta_j) \right) \log{q_i(\theta_i)} \, d\theta = \int \left(\prod_{j\neq i} q_j(\theta_j) \right) q_i(\theta_i) \log{q_i(\theta_i)} \, d\theta \\ = \left( \int q_i(\theta_i) \log{q_i(\theta_i)} \, d\theta_i \right) \left( \prod_{j \neq i} \int q_j(\theta_j) \, d\theta_j \right). $$ The integrals in the second bracket are all $1$, and we get the answer.

$\endgroup$
2
  • $\begingroup$ Thank you, very illuminating answer. $\endgroup$
    – tomka
    May 26, 2017 at 22:08
  • $\begingroup$ Didn't notice your answer when I posted. Very nice. $\endgroup$
    – Alex Ortiz
    May 26, 2017 at 23:02
1
$\begingroup$

It will probably suffice to see it worked out when $D = 2$:

\begin{align*} \int q_1q_2\ln(q_1q_2) & = \int q_1q_2(\ln q_1+\ln q_2) \\ &=\int q_1q_2\ln q_1 + \int q_1q_2\ln q_2. \end{align*}

Now the first integral is separable: \begin{align*} \iint q_1(\theta_1)q_2(\theta_2)\ln(q_1(\theta_1))\,d\theta_1\,d\theta_2&= \int q_2(\theta_2)\,d\theta_2\cdot\int q_1(\theta_1)\ln(q_1(\theta_1))\,d\theta_1. \end{align*} And likewise the second is also separable: \begin{align*} \iint q_1(\theta_1)q_2(\theta_2)\ln(q_2(\theta_2))\,d\theta_1\,d\theta_2&= \int q_1(\theta_1)\,d\theta_1\cdot\int q_2(\theta_2)\ln(q_2(\theta_2))\,d\theta_2. \end{align*} Since $\int q_i = 1$, you know how to take it from here.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .