Proving Renyi entropy properties

Question

I want to prove that Shannon entropy is a special case of Renyi entropy by solving this, $$ \lim_{\alpha\to1}\frac{1}{1-\alpha}\ln\sum_{k=1}^{n}p_{k}^{\alpha} = -\sum_{k}p_{k}\ln p_{k} $$

Answer 1

This is a quick exercise in L'Hôpital's rule:

\begin{align} &\lim_{\alpha \to 1} \frac{\ln\left(\sum p_i^\alpha\right)}{1 - \alpha} \tag{1}\\ \overset{(a)}= ~& \lim_{\alpha \to 1} \frac{ \left(\sum p_i^\alpha\right)^{-1} \sum p_i^{\alpha}\ln p_i}{-1} \\ = ~&\frac{-\sum p_i \ln p_i}{\sum p_i} = -\sum p_i \ln p_i \end{align}

Where $(a)$ follows from noting the limit $(1)$ is in the $\frac{0}{0}$ form as $\sum p_i = 1$. The rest is simple when you recall that $\frac{\mathrm{d}}{\mathrm{d}x} a^x = a^x \ln a.$

You should note, btw, that most of the time both Renyi and Shannon entropies are defined using $\log_2$ instead of $\ln$. This doesn't really alter anything, as both sides can be converted to the appropriate base by multiplying with a constant, and the above goes through with no change.