Show that $KL_X(\theta;\psi) = (\theta-\psi)^2/2$. Suppose that $Q$ and $P$ are two probability measures with densities $p$ and $q$ w.r.t. a common measure $\nu$.
The Kullback-Leibler information of $Q$ from $P$ is defined by $$KL(\theta\,;\psi) = \int\log\dfrac{p(x)}{q(x)}p(x)d\nu(x).$$ In case of parametric families, where $X\sim P_\theta$ and $P_\theta \ll \nu$ for all $\theta$ with densities $f_X(\cdot|\theta)$, we write $$KL(\theta\,;\psi) = \operatorname{E}_\theta\log\dfrac{f_X(X|\theta)}{f_X(X|\psi)}, \,\,\theta,\psi \in \Omega.$$ 
Exercise: If $X\sim N(\theta,1)$, show that $KL_X(\theta\,,\psi) = (\theta - \psi)^2/2.$
What I've tried to do: If I'm not mistaken the normal distribution is part of the exponential family, and I need to use the second part of the definition. I get $KL_X(\theta\,,\psi) = \operatorname{E}_\theta\dfrac{(x-\theta)^2}{(x-\psi)^2}$, which would mean that $KL_X(\theta\,,\psi) = 0$.
Questions:


*

*I'm not sure whether I fully grasp the concept of probability measures and densities. This definition states that the probability measures have a density of their own. But don't we speak of densities of distributions with respect to their probability measures? Furthermore, what would a common measure $\nu$ be? What would the probability measures in this exercise be?

*How do I solve this exercise?
 A: We have that $KL(\theta\,;\psi) = \operatorname{E}_\theta\log\dfrac{f_{X|\theta}}{f_{X|\psi}}$, and we know that for $X|\theta \sim N(\theta,1)$ the density is given by: 
\begin{equation}
f_{X|\theta} = \frac{\textrm{Exp}[-\frac{1}{2}(x-\theta)^2]}{\sqrt{2\pi}}
\end{equation}
Then the log of the ratio of the two densities is then:
\begin{equation}
\begin{split}
\textrm{log}\frac{f_{X|\theta}}{f_{X|\psi}} & = \textrm{log}f_{X|\theta} - \textrm{log}f_{X|\psi} \\
& = \frac{1}{2}[-(x-\theta)^2 + (x-\psi)^2]\\
& = -\frac{1}{2}[-x^2 + 2x\theta -\theta^2 + x^2 - 2x\psi +\psi^2]
\end{split}
\end{equation}
Taking the expectation with the distribution under $\theta$, we have $E[x] = \theta$ and $E[x^2] = 1+\theta^2$.
Then:
\begin{equation}
\begin{split}
-\frac{1}{2}[- x^2 + 2x\theta -\theta^2 + x^2 - 2x\psi +\psi^2] & = -\frac{1}{2}[-1-\theta^2 + 2\theta^2 -\theta^2 + 1 + \theta^2 - 2\theta\psi + \psi^2]\\
& = \frac{1}{2}[\theta^2 -2\theta\psi +\psi^2]\\
& = \frac{(\theta-\psi)^2}{2}
\end{split}
\end{equation}
