1
$\begingroup$

I want to prove that Shannon entropy is a special case of Renyi entropy by solving this, $$ \lim_{\alpha\to1}\frac{1}{1-\alpha}\ln\sum_{k=1}^{n}p_{k}^{\alpha} = -\sum_{k}p_{k}\ln p_{k} $$

$\endgroup$

1 Answer 1

6
$\begingroup$

This is a quick exercise in L'Hôpital's rule:

\begin{align} &\lim_{\alpha \to 1} \frac{\ln\left(\sum p_i^\alpha\right)}{1 - \alpha} \tag{1}\\ \overset{(a)}= ~& \lim_{\alpha \to 1} \frac{ \left(\sum p_i^\alpha\right)^{-1} \sum p_i^{\alpha}\ln p_i}{-1} \\ = ~&\frac{-\sum p_i \ln p_i}{\sum p_i} = -\sum p_i \ln p_i \end{align}

Where $(a)$ follows from noting the limit $(1)$ is in the $\frac{0}{0}$ form as $\sum p_i = 1$. The rest is simple when you recall that $\frac{\mathrm{d}}{\mathrm{d}x} a^x = a^x \ln a.$

You should note, btw, that most of the time both Renyi and Shannon entropies are defined using $\log_2$ instead of $\ln$. This doesn't really alter anything, as both sides can be converted to the appropriate base by multiplying with a constant, and the above goes through with no change.

$\endgroup$
1
  • 1
    $\begingroup$ Thank you very much. So, I guess for continuous variables when summation becomes integral, it is similar to this one also. $\endgroup$
    – TBBT
    Nov 19, 2015 at 4:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.