1
$\begingroup$

Let $p(x,\theta)$ be a density function and the KL-divergence be given by $$K\left(\theta,\theta^{'}\right)=\int \log \left( \frac{p(x,\theta)}{p(x,\theta^{'})} \right) p(x,\theta)dx$$ Let the fisher information be given by $$I\left(\theta\right)=\int \left\{\frac{\partial}{\partial\theta}\log p(x,\theta)\right\}^2 p(x,\theta)dx$$ To prove that

$$\frac{d^2}{d\theta^{'2}}K\left(\theta,\theta^{'}\right)|_{\theta^{'}=\theta}=I(\theta)$$

I'm trying to solve it but the left hand side always comes out to be $0$ due to miscalculation. Any help would be appreciated. Thank you.

$\endgroup$
2
$\begingroup$

We have $$ \partial_{\theta'} K(\theta,\theta') = \int -\frac{1}{p(x,\theta')}\partial_{\theta'}p(x,\theta') p(x,\theta)dx \\ \partial^2_{\theta'} K(\theta,\theta') =\int\left(\frac{1}{p(x,\theta')^2}(\partial_{\theta'}p(x,\theta'))^2 -\frac{1}{p(x,\theta')}\partial^2_{\theta'}p(x,\theta')\right)p(x,\theta)dx \\\to_{\theta'\to\theta}\int (\partial_{\theta}\log(p(x,\theta)))^2p(x,\theta)dx - \partial^2_{\theta}\int p(x,\theta)dx\\ = \int (\partial_{\theta}\log(p(x,\theta)))^2p(x,\theta)dx -0$$ where the last term is zero since $\int p(x,\theta)dx = 1.$

$\endgroup$
  • $\begingroup$ What's the justification of differentiating under the integral sign here? $\endgroup$ – Nap D. Lover Nov 12 '17 at 14:24
  • 1
    $\begingroup$ @LoveTooNap29 unspoken regularity conditions on $p(x,\theta).$ $\endgroup$ – spaceisdarkgreen Nov 12 '17 at 20:27
  • $\begingroup$ Hang on! isn't $\int p(x,\theta)dx = p(\theta)$ not $1$. $\endgroup$ – piccolo Jul 30 at 8:23
  • 1
    $\begingroup$ @piccolo $\theta$ is a parameter. $\endgroup$ – spaceisdarkgreen Jul 30 at 12:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.