This is from the bok "Pattern Recognition and Machine Learning" By Bishop. I am having a hard time following the last step of this equation Where Stirlings approxaimation is subtituted for $\ln N!$ to arrive at the final step...

We can understand this alternative view of entropy by considering a set of N identical objects that are to be divided amongst a set of bins, such that there are $n_i$ objects in the $i_{th}$ bin. Consider

the number of different ways of allocating the objects to the bins. There are $N$ ways to choose the first object, $(N − 1)$ ways to choose the second object, and so on, leading to a total of $N!$ ways to allocate all $N$ objects to the bins, where $N!$ (pronounced ‘factorial N ’) denotes the product $N × (N − 1) × · · · × 2 × 1$. However, we don’t wish to distinguish between rearrangements of objects within each bin. In the $i^th$ bin there are $n_i!$ ways of reordering the objects, and so the total number of ways of allocating the $N$ objects to the bins is given by

$$ W = \frac{N!}{\prod_i n_i!} $$

which is called the multiplicity. The entropy is then defined as the logarithm of the multiplicity scaled by an appropriate constant

$$ H = \frac{1}{N} \ln W = \frac{1}{N} \ln N! - \frac{1}{N} \sum_i \ln n_i! $$

We now consider the limit $N → ∞$, in which the fractions $n_i/N$ are held fixed, and apply Stirlings approximation

$$ \ln N! \backsimeq N \ln N - N $$

which gives

$$ H = - \lim_{x \rightarrow \infty} \sum_i \Big( \frac{n_i}{N} \Big) \ln \Big( \frac{n_i}{N} \Big) = - \sum_i p_i \ln p_i $$

where we have used $\sum_i n_i = N$. Here $p_i = \lim_{N \rightarrow \infty} (ni/N)$ is the probability of an object being assigned to the $i_{th}$ bin. In physics terminology, the specific ar- rangements of objects in the bins is called a microstate, and the overall distribution of occupation numbers, expressed through the ratios $n_i/N$, is called a macrostate. The multiplicity $W$ is also known as the weight of the macrostate.

  • 2
    $\begingroup$ Comments here should be helpful: math.stackexchange.com/questions/907056/… . Basically use Stirling's approximation on $\textit{both}$ $\ln N$ and $\ln n_i$, $\endgroup$ – twnly May 19 at 0:18
  • $\begingroup$ you meant to have a factorial in your comment next to $ln N$ and $n_i$, right? $\endgroup$ – deltaskelta May 19 at 0:25
  • $\begingroup$ Yep, my bad. Just apply Stirling's formula to both quantities and simplify. $\endgroup$ – twnly May 19 at 0:28
  • $\begingroup$ I'm still having some trouble simplifying it down right. How should it go? $\endgroup$ – deltaskelta May 19 at 0:42

After doing the substitution for $\ln N!$ and $\ln n_i!$ in the definition of $H$, and remembering that $\sum_i n_i = N$, you should have come up with the following equation:

$$ H \approx \ln N - \frac{1}{N}\sum_i n_i \ln n_i $$

The trick then is to multiply the $\ln N$ by 1, in the form $\frac{\sum_i n_i}{N}$. The result then follows directly: $$ H \approx \frac{\sum_i n_i}{N} \ln N - \frac{1}{N}\sum_i n_i \ln n_i = -\frac{1}{N}\sum_i n_i (\ln n_i - \ln N) = -\sum_i \frac{n_i}{N} \ln \left ( \frac{n_i}{N}\right ) $$

  • $\begingroup$ Thanks for your answer. I did arrive at your first point and the multiplication by 1 is what I was missing. I am fuzzy on the first step of your second equation though. How do you get from the beginning to having the $n_i(\ln n_i - \ln N)$ as the summand? What property of logarithms is being used there? $\endgroup$ – deltaskelta May 19 at 3:09
  • 1
    $\begingroup$ Oh nvm my comment, I see it is just a distribution into the sums now $\endgroup$ – deltaskelta May 19 at 12:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.