# Variational Lower Bound on marginal log-likelihood

I'm currently trying to understand the math behind the publication of Xu et al. (Neural Image Caption Generation with Visual Attention).

In this publication they define an objective function $L_s$ which is a variational lower bound on the marginal log-likelihood $p(y|a)$ of observing a sequence of words y given image features a.

$L_s = \sum_s p(s|a) log[ p(y|s,a)] \leq log [\sum_s p(s|a)p(y|s,a)] = log [p(y|a)]$ (equation 10)

The objective is to derive parameters W (which are no further specified) by directly optimizing $L_s$

Can someone explain how the following equation was obtained? p(y|s,a) and p(s|a) are no furhter specified.

$\frac{\partial L_s}{\partial W} = \sum_s p(s|a)\left[ \frac{\partial log[p(y|s,a)]}{\partial W} + log[p(y|s,a)] \frac{\partial log[p(s|a)]}{\partial W} \right]$

Published literature is not always the best place to find all the terms fully explained.

I haven't read the paper but presumably the authors are aiming to perform inference on the parameters $a$ by considering the distribution $p(y|a)$, now for whatever reason this distribution is presumably intractable in it's given form, but if we introduce some latent variables $s \in \mathcal{S}$ and the vector $W$ is some instance of $s$,

Now since $a$ is kept fixed throughout this I am going to drop it from my notation and then reintroduce it at the end, now conventional variational inference tells us that \begin{align*} \ln p(y) = \mathcal{L} + D_{KL}, \end{align*} where \begin{align*} \mathcal{L} = \sum_s q(s) \log \frac{ p(y ,s) }{q(s)} \end{align*} and \begin{align*} D_{KL} = -\sum_s q(s) \log \frac{ p(s|y) }{q(s)}. \end{align*} So that using \begin{align*} p(y,s) = p(y|s)q(s), \end{align*} we have \begin{align*} \ln p(y) &= \sum_s q(s) \log \frac{p(y,s)}{q(s)}+ D_{KL} \\ &= \sum_s q(s) \log \frac{ p(y|s)q(s) }{q(s)} + D_{KL}\\ &= \sum_s q(s) \log p(y | s) + D_{KL} \\ &= L_s + D_{KL} \end{align*} Now since $D_{KL} \geq 0$ we have $L_s \leq \log p(y)$ which is the sense in which it is a "lower bound" on the log probability. To complete the conversion to their notation just add the additional conditional dependence on $a$.

Now to maximise the marginal log-likelihood for a fixed value of $a$ we can proceed to try and make $L_s$ as large as possible. Now I'm finding their notation a bit clunky - perhaps it is clearer in the article, but it seems $W$ is just a particular instance of the state of $s$, but anyway we have using the product rule and switching to the logarithmic derivative \begin{align*} \frac{\partial}{\partial W}L_s &= \sum_s \frac{\partial p(s|a)}{\partial W}\log p(y | s , a) + p(s|a) \frac{\partial p(y | s ,a)}{\partial W} \\ &= \sum_s \left( p(s | a)\frac{\partial \log p(s|a)}{\partial W} \right)\log p(y|s , a) + p(s |a)\frac{\partial \log p(y |s,a)}{\partial W} \\ &= \sum_s p(s|a) \left[ \frac{\partial \log p(s|a)}{\partial W}\log p(y| s,a) + \frac{\partial \log p(y| s, a )}{\partial W} \right] \end{align*}

• That's a really good answer! Thanks! Can you recommend some good literature about variational inference? – Cr3at0rX Feb 28 '17 at 20:35
• You are very welcome. There are quite a lot of lecture notes online that describe the method, in particular in connection with the EM algorithm which can be see as a particular instance of maximising the different bounds so try googling that. If you want a text then Bishop's "Pattern Recognition and Machine Learning" introduces this material in a really accessible way – Nadiels Feb 28 '17 at 21:34
• @Cr3at0rX If you think it's a really good answer (as I do) upvote it (as I did) in addition to accepting it! – Anonymous Feb 28 '17 at 22:29
• @Anonymous I would like to upvote his answer, but my reputation is not high enough to do so. – Cr3at0rX Mar 1 '17 at 10:37
• @Anonymous, upvoted, very good answer. While I didn't understand 100% of the steps, Nadiels' answer helped me to get closer to grasping the concept. The wiki on deriving the variational inference equation helps as well, although the KL is different – vega Sep 5 '17 at 15:23