# computing variational lower bound

I am trying to re-derive the variational lower bound for the binary logistic regression that is obtained in the paper by Saul & Jordan 1999 and it is given in equation 22 $$\langle \ln(1+e^z)\rangle\leq\frac{1}{2}\xi^2\langle\delta z^2\rangle+\ln\Big(1+e^{\langle z\rangle+(1-2\xi)\langle\delta z^2\rangle/2)}\Big)$$

Could anybody suggest how I can get from equation 21 to 22?

In a similar topic, in a paper by Knowles and Minka 2011 they found another bound which is given in section 5.1 (in the third paragraph) which is again not clear how it is computed. Any suggestion or thought? Thanks

Using the moment generating function of the normal distribution, given by $$\langle e^{tz}\rangle = e^{t\langle z \rangle + \tfrac{1}{2}\langle \delta z^2\rangle t^2}$$, and the linearity of expectation we can write $$\langle e^{-\xi z} + e^{(1-\xi)z} \rangle = \langle e^{-\xi z} \rangle + \langle e^{(1-\xi)z} \rangle = e^{-\xi\langle z \rangle + \tfrac{1}{2}\langle \delta z^2\rangle \xi^2} + e^{(1-\xi)\langle z \rangle + \tfrac{1}{2}\langle \delta z^2\rangle (1-\xi)^2} \\ = e^{-\xi\langle z \rangle + \tfrac{1}{2}\langle \delta z^2\rangle \xi^2}(1 + e^{\langle z \rangle + \tfrac{1}{2}\langle \delta z^2\rangle (1-2\xi)}).$$ From here on you should be able to work out the term $$\xi\langle z\rangle + \ln\langle e^{-\xi z} + e^{(1-\xi)z} \rangle$$.
• thanks for answering the first part of my question but I do not see a link between the slides of Jordan with the derivation in Knowles paper.$S(m,v)=\mu_T\langle \mathbf{u}(x)\rangle_{\mathcal{N(m,v)}}-c$ and I can not understand how for logistic regression we have $\frac{dS}{dv}=-\frac{\langle x\sigma(x)\rangle_q-m\langle \sigma(x)\rangle_q}{2v}$ and $\frac{dS}{dv}=s-\langle \sigma(x)\rangle_q$? any suggestion? Thanks again! – Dalek May 5 at 18:01
• I assumed that you want to know about equation (8) of the Knowles paper, is that correct? This bound seems to be related to the variational identity $\sigma(x) = \min_{\lambda}[e^{\lambda x + \lambda \log \lambda + (1-\lambda) \log(1-\lambda)}]$ (taken from the slides). The bound in equation (9) is a direct consequence of the bound of Saul & Jordan 1999. The derivatives you just mentioned are a result of a completely different technique (quadrature) with which I'm unfamiliar, have nothing to do with bounds (8) and (9), and on which I'd suggest to post a new question. – pabk May 7 at 15:04