1
$\begingroup$

This is a question similar to 1 and 2. I wan to find out what the relationship between Kullback-Leibler divergence and L2 norm. Finally, I get the answer in one book named "Smoothing of Multivariate Data: Density Estimation and Visualization". But I can not prove the Lemma in this book. Does anyone can help me to prove the Lemma together. And my probability density function is a GMM. Thanks in advance.

For the sake of completeness I quote the Lemma 11.6:

Let $D_k$ be the Kullback-Leibler distance defined in (11.8). We have $$D_k^2(f,f_0)\le\int_{\{x\in \mathbb{R}^d:f_0(x)\gt0\}}\frac{(f-f_0)^2}{f_0}.$$ The upper bound is called the $\chi^2$-divergence between $f$ and $f_0$. In particular if $\inf_{x\in \mathbb{R}^d}$$f_0(x)\gt0$, then $$D_k^2(f,f_0)\le\frac{\lVert f-f_0\rVert_2^2}{\inf_{x\in \mathbb{R}^d}f_0(x)}.$$ Also, if $f$ and $f_0$ are both bounded and bounded away from zero, then $$D_k^2(f,f_0)\ge\int_{\{x\in \mathbb{R}^d:f_0(x)\gt0\}}(f-f_0)+C\lVert f-f_0\rVert_2^2,$$ for a positive constant C.

Some other formulas maybe used are given as follows:$$D_k^2(f,g)=\int_{\mathbb{R}^d\cap\{x:g(x)\gt0\}}f\log_e(\frac{f}{g})\quad\quad(11.8)$$ $$\lVert f-\hat f\rVert_P^P=\int_{R^d}\lvert f-\hat f\rvert^P$$

$\endgroup$
  • $\begingroup$ We will need to linearize the effect of the logarithm somehow. $\endgroup$ – mathreadler Feb 4 '18 at 9:10
  • $\begingroup$ taylor expansion? $\endgroup$ – Peter.J Feb 4 '18 at 11:16
  • $\begingroup$ I don't see how that makes it linear. $\endgroup$ – mathreadler Feb 4 '18 at 12:24
  • $\begingroup$ In the post "math.stackexchange.com/questions/2614201/…", taylor expansion is used. But Lemma11.6 is different from theirs question. $\endgroup$ – Peter.J Feb 4 '18 at 12:34
  • $\begingroup$ Are you sure that it's not supposed to be $f$ in the denominator in the first expression? Also is $D^2_k$ the KL divergence squared, or the KL divergence by itself? $\endgroup$ – Ryan Warnick Feb 5 '18 at 7:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.