Is there a conditional version of the asymptotic equipartition property?

Let $X_i$ be independent random variables with $\operatorname{Pr}(X_i = x) = p_x$, and let $F_n$ be the empirical frequency distribution of $X_1, \ldots, X_n$: that is, $(F_n)_x$

For any frequency distribution $f$, write $K(f \| p)$ for the relative entropy $$\sum_x f_x \log \frac{f_x}{p_x}.$$ As $n$ grows, $K(F_n \| p)$ converges in probability to $K(p \| p)$, which is zero. This is basically the asymptotic equipartition property.

Now, condition on $F_n$ lying in some open set $U$ in the space of frequency distributions. Conditionally, I suspect that $K(F_n \| p)$ should converge in probability to $\inf_{f \in U} K(f \| p)$. Is this true? (Maybe with extra assumptions?) If so, how do you prove it?

• Good question. But, is that called asymptotic equipartition property? Commented May 24, 2013 at 4:47
• This kind of asymptotics is what large deviations theory was invented for. The first chapter of Dembo and Zeitouni's treatise explains this.
– Did
Commented May 24, 2013 at 5:34