Random variables $X_1,X_2,\cdots,X_n$ have joint probability distribution $\mathbb{Q}_{X_1,X_2,\cdots,X_n}(x_1,x_2,\cdots,x_n)$. Assume another probability function $\mathbb{P}_Z(z)$. What does $\lim_{n \to \infty}\frac{1}{n} \log \frac{\mathbb{P}_{Z}(x_1) \cdots \mathbb{P}_{Z}(x_n)}{\mathbb{Q}_{X_1,\cdots,X_n}(x_1,\cdots,x_n)}$ converge to?

I don't know the answer for the general $\mathbb{Q}$, but if we assume $X_i$s are independent and identically distributed,$\mathbb{Q}_{X_1,X_2,\cdots,X_n}(x_1,x_2,\cdots,x_n) = \mathbb{Q}(x_1) \cdots \mathbb{Q}(x_n)$. Therefore

$\lim_{n \to \infty}\frac{1}{n} \log \frac{\mathbb{P}_{Z}(x_1) \cdots \mathbb{P}_{Z}(x_n)}{\mathbb{Q}_{X_1,\cdots,X_n}(x_1,\cdots,x_n)} = \lim_{n \to \infty}\frac{1}{n}\log\frac{\mathbb{P}_{Z}(x_1) \cdots \mathbb{P}_{Z}(x_n)}{\mathbb{Q}(x_1)\cdots\mathbb{Q}(x_n)} = \lim_{n \to \infty}\frac{1}{n}\sum_{n=1}^{\infty} \log \frac{\mathbb{P}_{Z}(x_1)}{\mathbb{Q}(x_1)}$

Using the law of large number, we see that the above is equal to $\mathbb{E}_{Q}[\frac{P_Z(x)}{Q(x)}] = - \mathbb{D}_{KL}(Q(x)||P_Z(x))$.

Can we obtain a similar result for general $\mathbb{Q}$ distribution? Please let me know of any theorem or lemma that helps.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.