0
$\begingroup$

I am trying to re-derive the variational lower bound for the binary logistic regression that is obtained in the paper by Saul & Jordan 1999 and it is given in equation 22 $$\langle \ln(1+e^z)\rangle\leq\frac{1}{2}\xi^2\langle\delta z^2\rangle+\ln\Big(1+e^{\langle z\rangle+(1-2\xi)\langle\delta z^2\rangle/2)}\Big)$$

Could anybody suggest how I can get from equation 21 to 22?

In a similar topic, in a paper by Knowles and Minka 2011 they found another bound which is given in section 5.1 (in the third paragraph) which is again not clear how it is computed. Any suggestion or thought? Thanks

$\endgroup$
1
$\begingroup$

Using the moment generating function of the normal distribution, given by $\langle e^{tz}\rangle = e^{t\langle z \rangle + \tfrac{1}{2}\langle \delta z^2\rangle t^2}$, and the linearity of expectation we can write $$ \langle e^{-\xi z} + e^{(1-\xi)z} \rangle = \langle e^{-\xi z} \rangle + \langle e^{(1-\xi)z} \rangle = e^{-\xi\langle z \rangle + \tfrac{1}{2}\langle \delta z^2\rangle \xi^2} + e^{(1-\xi)\langle z \rangle + \tfrac{1}{2}\langle \delta z^2\rangle (1-\xi)^2} \\ = e^{-\xi\langle z \rangle + \tfrac{1}{2}\langle \delta z^2\rangle \xi^2}(1 + e^{\langle z \rangle + \tfrac{1}{2}\langle \delta z^2\rangle (1-2\xi)}).$$ From here on you should be able to work out the term $\xi\langle z\rangle + \ln\langle e^{-\xi z} + e^{(1-\xi)z} \rangle$.

$\endgroup$
  • $\begingroup$ Thanks for the answer! Can the expectation of sum be equal to the sum of expectations? Do you also have a clue how the lower bound in the paper by knowles et al 2011 has been calculated? Thanks again :) $\endgroup$ – Dalek May 4 at 17:37
  • $\begingroup$ For your first question, I'd refer to this Wikipedia entry. It also simply follows from the linearity of integrals. I know very little of calculus of variations, so I don't have a definitive answer for the lower bound. It seems that if you work out the expression for the sigmoid in this presentation (slide 7) (from the same author as Reference [9] from your paper) you should be able to get close. $\endgroup$ – pabk May 4 at 20:27
  • $\begingroup$ thanks for answering the first part of my question but I do not see a link between the slides of Jordan with the derivation in Knowles paper.$S(m,v)=\mu_T\langle \mathbf{u}(x)\rangle_{\mathcal{N(m,v)}}-c$ and I can not understand how for logistic regression we have $\frac{dS}{dv}=-\frac{\langle x\sigma(x)\rangle_q-m\langle \sigma(x)\rangle_q}{2v}$ and $\frac{dS}{dv}=s-\langle \sigma(x)\rangle_q$? any suggestion? Thanks again! $\endgroup$ – Dalek May 5 at 18:01
  • $\begingroup$ I assumed that you want to know about equation (8) of the Knowles paper, is that correct? This bound seems to be related to the variational identity $\sigma(x) = \min_{\lambda}[e^{\lambda x + \lambda \log \lambda + (1-\lambda) \log(1-\lambda)}]$ (taken from the slides). The bound in equation (9) is a direct consequence of the bound of Saul & Jordan 1999. The derivatives you just mentioned are a result of a completely different technique (quadrature) with which I'm unfamiliar, have nothing to do with bounds (8) and (9), and on which I'd suggest to post a new question. $\endgroup$ – pabk May 7 at 15:04
  • $\begingroup$ I was wondering how the bound they mentioned in the text of section 5.1 was computed! I could find the derivation of equation 8 in Jordan paper and thanks to your answer, the proof for equation 9 was clarified. $\endgroup$ – Dalek May 7 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.