# Relationship between Fisher information and KL-divergence

Let $p(x,\theta)$ be a density function and the KL-divergence be given by $$K\left(\theta,\theta^{'}\right)=\int \log \left( \frac{p(x,\theta)}{p(x,\theta^{'})} \right) p(x,\theta)dx$$ Let the fisher information be given by $$I\left(\theta\right)=\int \left\{\frac{\partial}{\partial\theta}\log p(x,\theta)\right\}^2 p(x,\theta)dx$$ To prove that

$$\frac{d^2}{d\theta^{'2}}K\left(\theta,\theta^{'}\right)|_{\theta^{'}=\theta}=I(\theta)$$

I'm trying to solve it but the left hand side always comes out to be $0$ due to miscalculation. Any help would be appreciated. Thank you.

## 1 Answer

We have $$\partial_{\theta'} K(\theta,\theta') = \int -\frac{1}{p(x,\theta')}\partial_{\theta'}p(x,\theta') p(x,\theta)dx \\ \partial^2_{\theta'} K(\theta,\theta') =\int\left(\frac{1}{p(x,\theta')^2}(\partial_{\theta'}p(x,\theta'))^2 -\frac{1}{p(x,\theta')}\partial^2_{\theta'}p(x,\theta')\right)p(x,\theta)dx \\\to_{\theta'\to\theta}\int (\partial_{\theta}\log(p(x,\theta)))^2p(x,\theta)dx - \partial^2_{\theta}\int p(x,\theta)dx\\ = \int (\partial_{\theta}\log(p(x,\theta)))^2p(x,\theta)dx -0$$ where the last term is zero since $\int p(x,\theta)dx = 1.$

• What's the justification of differentiating under the integral sign here? – Nap D. Lover Nov 12 '17 at 14:24
• @LoveTooNap29 unspoken regularity conditions on $p(x,\theta).$ – spaceisdarkgreen Nov 12 '17 at 20:27
• Hang on! isn't $\int p(x,\theta)dx = p(\theta)$ not $1$. – piccolo Jul 30 at 8:23
• @piccolo $\theta$ is a parameter. – spaceisdarkgreen Jul 30 at 12:45