Tag Info

New answers tagged


Here is a simple intuitive construction of Shannon's Entropy formula: Understanding Shannon's Entropy...


I suspect there's a well defined special function related to this series. Is it so ? Not really. Basically, $\text{Exl}(x)=-F'(1),$ where $F(k)=\displaystyle\sum_{n\ge0}\frac{x^n}{n!^k}~.~$ The only known values of F are $F(0)=\dfrac1{1-x},~F(1)=e^x,$ and $F(2)=I_0\big(2\sqrt x\big).$ See Bessel function for more information. Neither F, let alone ...


Hint. You may just observe that, as $N \to \infty$, we have $$ x^{\large\frac1N}=e^{\large\frac{\ln x}N}=1+\frac{\ln x}N+\frac{\ln^2 x}{2N^2}+O\left(\frac1{N^3}\right) $$ giving directly $$ N(x^\frac{1}{N}-1)\sim\ln x+\frac{\ln^2 x}{2N} $$ as announced. We have used the Taylor expansion $$ e^u=1+u+\frac{u^2}{2!}+O(u^3) $$ as $u \to 0$.


The derivative of $x\mapsto x\ln x$ is $x\mapsto 1+\ln x$. So when you compute the functional derivative of $F[g]=\int_{-\infty}^{\infty}g(x)\ln(g(x))\mathrm dx$ with respect to $g$, you compute $$\delta F[g]=\int_{-\infty}^{\infty}\left[\left(g(x)+\delta g(x)\right)\ln(g(x)+\delta g(x))-g(x)\ln(g(x))\right]\mathrm ...


Log2 1/p is the number of bits needed to transmit symbols that occur with probability p. For example, if it occurs 1 times in 8, we need 3 bits to encode all 8 possibilities. Now just take the average number of bits weighted by p for each symbol.


Fisher information is related to the asymptotic variability of a maximum likelihood estimator. The idea being that higher Fisher Information is associated with lower estimation error. Shannon Information is totally different, and refers to the content of the message or distribution, not its variability. Higher entropy distributions are assumed to convey ...


It's important to distinguish $H(X \mid Y)$ from $H(X \mid Y=y)$. The first is a number, the second is a function of $y$. The first is the average of the second -averaged over $p(y)$: $$H(X \mid Y) = E_y \left[H(X \mid Y=y)\right]$$ Hence, beacuse it's an average, in general we'll have $H(X \mid Y=y)>H(X \mid Y)$ for some values of $y$ and $H(X \mid ...

Top 50 recent answers are included