1
$\begingroup$

Random variables $X_1,X_2,\cdots,X_n$ have joint probability distribution $\mathbb{Q}_{X_1,X_2,\cdots,X_n}(x_1,x_2,\cdots,x_n)$. Assume another probability function $\mathbb{P}_Z(z)$. What does $\lim_{n \to \infty}\frac{1}{n} \log \frac{\mathbb{P}_{Z}(x_1) \cdots \mathbb{P}_{Z}(x_n)}{\mathbb{Q}_{X_1,\cdots,X_n}(x_1,\cdots,x_n)}$ converge to?

I don't know the answer for the general $\mathbb{Q}$, but if we assume $X_i$s are independent and identically distributed,$\mathbb{Q}_{X_1,X_2,\cdots,X_n}(x_1,x_2,\cdots,x_n) = \mathbb{Q}(x_1) \cdots \mathbb{Q}(x_n)$. Therefore

$\lim_{n \to \infty}\frac{1}{n} \log \frac{\mathbb{P}_{Z}(x_1) \cdots \mathbb{P}_{Z}(x_n)}{\mathbb{Q}_{X_1,\cdots,X_n}(x_1,\cdots,x_n)} = \lim_{n \to \infty}\frac{1}{n}\log\frac{\mathbb{P}_{Z}(x_1) \cdots \mathbb{P}_{Z}(x_n)}{\mathbb{Q}(x_1)\cdots\mathbb{Q}(x_n)} = \lim_{n \to \infty}\frac{1}{n}\sum_{n=1}^{\infty} \log \frac{\mathbb{P}_{Z}(x_1)}{\mathbb{Q}(x_1)}$

Using the law of large number, we see that the above is equal to $\mathbb{E}_{Q}[\frac{P_Z(x)}{Q(x)}] = - \mathbb{D}_{KL}(Q(x)||P_Z(x))$.

Can we obtain a similar result for general $\mathbb{Q}$ distribution? Please let me know of any theorem or lemma that helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.