Kullback–Leibler divergence between two parametrized distributions is defined as:

$$ D_{KL}(q(\theta) || p(\theta)) = \int q(\theta) \log \frac{q(\theta)}{p(\theta)} d\theta $$

Rényi divergence is defined as:

$$ D_{\alpha}(q(\theta) || p(\theta)) = \frac{1}{\alpha-1} \log\int p(\theta)^\alpha q(\theta)^{1-\alpha} d\theta $$

It is known that the KL divergence is a particular case of Rényi divergence when $\alpha \rightarrow 1$.

But what is the proof for that?


Take the limit with L'Hopital: \begin{align*} \lim_{\alpha\rightarrow 1} D_\alpha(p||q) &= \lim_{\alpha\rightarrow 1} \frac{1}{\alpha-1}\log\int p(\theta)^\alpha q(\theta)^{1-\alpha}\,d\theta \\ &= \lim_{\alpha\rightarrow 1} \frac{\partial}{\partial \alpha} \log\int p(\theta)^\alpha q(\theta)^{1-\alpha}\,d\theta \\[2mm] &= \lim_{\alpha\rightarrow 1} \frac{\displaystyle\int p(\theta)^\alpha q(\theta)^{1-\alpha} [\log(p(\theta))-\log(q(\theta))] \,d\theta}{\displaystyle \int p(\theta)^\alpha q(\theta)^{1-\alpha}\,d\theta} \\ &= \frac{\displaystyle \int p(\theta)\log(p(\theta)/q(\theta))\,d\theta}{\displaystyle \int p(\theta)\,d\theta} \\[2mm] &=\int p(\theta)\log\left(\frac{p(\theta)}{q(\theta)}\right)\,d\theta \\[2mm] &= D_{\text{KL}}(p||q) \end{align*}


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.