# MLE coincides asymptotically with true value of parameter (II)

I'm trying to understand the second part of a proof which was presented here before. Basically, we want to establish:

$$\lim_{n\to \infty} P_{\theta_0} [L(\theta _0, X) > L(\theta, X)]=1, \,\,\forall \theta \neq\theta _0$$

The following steps are already established:

$$\lim_{n\to \infty} P_{\theta_0} [L(\theta _0, X) > L(\theta, X)] = \lim_{n\to \infty} P_{\theta_0} \Big[\frac1n \sum_{i=1}^n \log \Big[\frac{f(X_i;\theta)}{f(X_i;\theta_0)}\Big] < 0\Big]$$

Using Jensen's inequality and the weak law of large numbers one establishes that $\frac1n \sum_{i=1}^n \log \Big[\frac{f(X_i;\theta)}{f(X_i;\theta_0)}\Big] \to_{P_{\theta_0}} E_{\theta_0} \log \Big[\frac{f(X_1;\theta)}{f(X_1;\theta_0)}\Big] < 0$. From here one concludes the wanted limit equality.

My questions

1. How do I know that the expected value of $\log \Big[\frac{f(X_1;\theta)}{f(X_1;\theta_0)}\Big]$ is finite? In the book by Kullback, Information Theory and Statistics I read that the Kullback-Leibler divergence always exists but it may be infinite.

2. How can I conclude from the last fact that the original hypothesis is true? Is it because I can permute the limit and the symbol $P$?

2. I write $\frac1n \sum_{i=1}^n \log \Big[\frac{f(X_i;\theta)}{f(X_i;\theta_0)}\Big] \to_{P_{\theta_0}} E_{\theta_0} \log \Big[\frac{f(X_1;\theta)}{f(X_1;\theta_0)}\Big]$ by definition of convergence in probability as
$$\forall \epsilon > 0. \lim_{n\to \infty} P_{\theta_0}\Big[\Big|\frac1n \sum_{i=1}^n \log \Big[\frac{f(X_i;\theta)}{f(X_i;\theta_0)}\Big]- E_{\theta_0} \log \Big[\frac{f(X_1;\theta)}{f(X_1;\theta_0)}\Big]\Big| < \epsilon\Big] = 1$$
and this says that the distance to a negative point is less than any $\epsilon$. I can take $\epsilon$ sufficiently small so that with probability 1 the average of the sum of logs is negative. It would be nice if someone dared to formalize this bit rigorously.
• $\frac1n \sum_{i=1}^n \log \Big[\frac{f(X_i;\theta)}{f(X_i;\theta_0)}\Big] < 0$ is asymptotically dirac delta at (-) KL divergence which is always positive. So your random variable will asymptotically be negative. Since this happens under $P_{\theta_0}$, i.e. the (-) of the related KL divergence is calculated w.r.t. $P_{\theta_0}$, the result follows. – Seyhmus Güngören Jan 5 '18 at 18:23
• Basically any $\frac1n \sum_{i=1}^n X_i\Big]$ is degenerate distribution, which i was calling as dirac delta density. So the question is where it lies. It lies at its expected value i.e. $E_{P_{\theta_0}}\log[f(\mathbf{X},\theta)/f(\mathbf{X},\theta_0)]=-D_{KL}(P_{\theta_0},P_\theta)$. Since $D_{KL}$ is always non-negative what you are after is positive unless $P_{\theta_0}=P_{\theta_0}$ – Seyhmus Güngören Jan 5 '18 at 19:09