Entropy of geometric random variable? I am wondering how to derive the entropy of a geometric random variable? Or where I can find some proof/derivation? I tried to search online, but seems not much resources is available.
Here is the probability density function of geometric distribution:
$(1 - p)^{k-1}\,p$
Here is the entropy of a geometric distribution:  $\frac{-(1-p)\log_2 (1-p) - p\log_2 p}{p}$
Where $p$ is the probability for the event to occur during each single experiment.
Thanks a lot.
 A: Assume $ P(X=k) = (1-p)^{k-1}p $, where $ k \in Z^{+} $, then the entropy is
$$ \begin{aligned}
Entropy(X) & = \sum_{k=1}^{+\infty} -(1-p)^{k-1}p \cdot \log_{2}{((1-p)^{k-1}p)} \\
& = -p \cdot \log_{2}{(p)} \sum_{k=1}^{+\infty} (1-p)^{k-1} - p \cdot \log_{2}{(1-p)} \sum_{k=1}^{+\infty} (k-1)(1-p)^{k-1} \\
& = - \log_{2}{(p)} - \frac{(1-p)\log_{2}{(1-p)}}{p}
\end{aligned} $$
A: It may be faster to derive the above entropy using the memoryless property of the Geometric distribution. Let $Y$ be an auxiliary binary random variable such that $Y=0$ indicates $X>1$. Then $H(X) = H_B(p)/p$ follows from the following:
\begin{align}
H(X, Y) &= H(X|Y) + H(Y)
\\
&= H(Y|X) + H(X)
\end{align}
Note that 
\begin{align}
H(X|Y) &= H(X|Y=1)P(Y=1) + H(X|Y=0)P(Y=0) = H(X|Y=0)(1-p) = (1-p)H(X)
\\
H(Y|X) &= 0
\\
H(Y) &= H_B(p),
\end{align}
where the memoryless property indicates $H(X|Y=0) = H(X)$, and $H_B(p)$ is the binary entropy function. 
A: Given: $X$ has Geometric PMF with probability $p$
$H(X) =  E(I(X)) = -Elog(P(X)) = -[E(log(p))+{log(1-p)}E(x-1)] = -[log(p)+\frac{1-p}{p}log(1-p)]$
Therefore, $H(X) = -\dfrac{plog(p)+(1-p)log(1-p)}{p}$
where $E(X)= \dfrac{1}{p}$
A: Let $ P(X=k) = (1-p)^{k-1}p $, where $ k \in Z^{+} $, then the entropy is given as
$$ \begin{aligned}
H(X) & = \sum_{k=1}^{+\infty} -(1-p)^{k-1}p \cdot \log_{2}{((1-p)^{k-1}p)} \\
& = -p \cdot \log_{2}{(p)} \sum_{k=1}^{+\infty} (1-p)^{k-1} - p \cdot \log_{2}{(1-p)} \sum_{k=1}^{+\infty} (k-1)(1-p)^{k-1} \\
\end{aligned} $$
Now, the first series is a straightforward infinite geometric progression sum, the second summation can be evaluated as follows
$$ \begin{aligned}
S = \sum_{k=1}^{+\infty} -(1-p)^{k-1}p \cdot \log_{2}{((1-p)^{k-1}p)}\\
S = -p \cdot \log_{2} (1-p) \cdot (0 +(1-p)+2(1-p)^2+3(1-p)^3 ....)\\
(1-p) \cdot S = -p \cdot \log_{2} (1-p) \cdot (0 +(1-p)^2+2(1-p)^3+3(1-p)^4 ....)\\
S -(1-p)S = -p \cdot \log_{2} (1-p) \cdot (0 +(1-p)+(1-p)^2+(1-p)^3 ....)\\
S \cdot p =  -p \cdot \log_{2} (1-p) \cdot ( \frac{1-p}{1 -(1-p)}) \\
S = -\log_{2} (1-p) \cdot ( \frac{1-p}{p}) 
\end{aligned} $$
Now, combining both the summations gives us,
$$\begin{aligned}
H(X) = -\log_{2}(p) - (\frac{1-p}{p})\log_{2}(1-p) 
\end{aligned}$$
