I'm trying to understand the second part of a proof which was presented here before. Basically, we want to establish:

$$ \lim_{n\to \infty} P_{\theta_0} [L(\theta _0, X) > L(\theta, X)]=1, \,\,\forall \theta \neq\theta _0$$

The following steps are already established:

$$ \lim_{n\to \infty} P_{\theta_0} [L(\theta _0, X) > L(\theta, X)] = \lim_{n\to \infty} P_{\theta_0} \Big[\frac1n \sum_{i=1}^n \log \Big[\frac{f(X_i;\theta)}{f(X_i;\theta_0)}\Big] < 0\Big] $$

Using Jensen's inequality and the weak law of large numbers one establishes that $ \frac1n \sum_{i=1}^n \log \Big[\frac{f(X_i;\theta)}{f(X_i;\theta_0)}\Big] \to_{P_{\theta_0}} E_{\theta_0} \log \Big[\frac{f(X_1;\theta)}{f(X_1;\theta_0)}\Big] < 0$. From here one concludes the wanted limit equality.

My questions

  1. How do I know that the expected value of $\log \Big[\frac{f(X_1;\theta)}{f(X_1;\theta_0)}\Big]$ is finite? In the book by Kullback, Information Theory and Statistics I read that the Kullback-Leibler divergence always exists but it may be infinite.

  2. How can I conclude from the last fact that the original hypothesis is true? Is it because I can permute the limit and the symbol $P$?

Towards the answer

  1. For the first part, it seems that the Kullback-Leibler divergence does the trick. But why is the Kullback-Leibler divergence always defined in my case?
  2. I write $ \frac1n \sum_{i=1}^n \log \Big[\frac{f(X_i;\theta)}{f(X_i;\theta_0)}\Big] \to_{P_{\theta_0}} E_{\theta_0} \log \Big[\frac{f(X_1;\theta)}{f(X_1;\theta_0)}\Big]$ by definition of convergence in probability as

$$\forall \epsilon > 0. \lim_{n\to \infty} P_{\theta_0}\Big[\Big|\frac1n \sum_{i=1}^n \log \Big[\frac{f(X_i;\theta)}{f(X_i;\theta_0)}\Big]- E_{\theta_0} \log \Big[\frac{f(X_1;\theta)}{f(X_1;\theta_0)}\Big]\Big| < \epsilon\Big] = 1$$

and this says that the distance to a negative point is less than any $\epsilon$. I can take $\epsilon$ sufficiently small so that with probability 1 the average of the sum of logs is negative. It would be nice if someone dared to formalize this bit rigorously.

  • $\begingroup$ $ \frac1n \sum_{i=1}^n \log \Big[\frac{f(X_i;\theta)}{f(X_i;\theta_0)}\Big] < 0$ is asymptotically dirac delta at (-) KL divergence which is always positive. So your random variable will asymptotically be negative. Since this happens under $P_{\theta_0}$, i.e. the (-) of the related KL divergence is calculated w.r.t. $P_{\theta_0}$, the result follows. $\endgroup$ – Seyhmus Güngören Jan 5 '18 at 18:23
  • 1
    $\begingroup$ yes, as I told you, because it is the negative of the KL divergence and the KL divergence is ALWAYS non-negative. If you dont know that KL divergence is just google it. $\endgroup$ – Seyhmus Güngören Jan 5 '18 at 19:00
  • $\begingroup$ Basically any $ \frac1n \sum_{i=1}^n X_i\Big]$ is degenerate distribution, which i was calling as dirac delta density. So the question is where it lies. It lies at its expected value i.e. $E_{P_{\theta_0}}\log[f(\mathbf{X},\theta)/f(\mathbf{X},\theta_0)]=-D_{KL}(P_{\theta_0},P_\theta)$. Since $D_{KL}$ is always non-negative what you are after is positive unless $P_{\theta_0}=P_{\theta_0}$ $\endgroup$ – Seyhmus Güngören Jan 5 '18 at 19:09
  • $\begingroup$ that result follows from the law of large numbers. Just check this one. or just see some proofs of it. There are very simple proof. en.wikipedia.org/wiki/Law_of_large_numbers#Weak_law. $\endgroup$ – Seyhmus Güngören Jan 5 '18 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.