In the paper VIME: Variational Information Maximizing Exploration, the authors suggest taking a single second-order step to efficiently optimize a variant of evidence lower bound

enter image description here

They say

To optimize Eq. (12) efficiently, we only take a single second-order step. This way, the gradient is rescaled according to the curvature of the KL divergence at the origin. As such, we compute $D_{KL} [q(\theta; \phi + \lambda\Delta\phi)\Vert q(\theta; \phi)]$, with the update step $\Delta\phi$ defined as $$ \Delta\phi=H^{-1}(l)\nabla_\phi l(q(\theta;\phi),s_t) \tag{13} $$ in which $H(l)$ is the Hessian of $l(q(\theta;\phi),s_t)$.

I'm quite confused about how they obtain Eq.$(13)$: why would they multiply the gradient by the inverse of the Hessian?

  • 1
    $\begingroup$ This is Newton’s method! $\endgroup$
    – David M.
    Mar 9 '19 at 7:00

I'm not familiar with the form of the cost function they are optimizing, but it seems to me that your question might be resolved by getting some intuition on second-order methods. See this wiki entry on a fundamental example and how second derivatives (i.e. Hessians) allow us to do better optimization by using curvature information of the function.

  • $\begingroup$ I see, thank you so much! $\endgroup$
    – Maybe
    Mar 9 '19 at 10:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.