Finding the mode of the negative binomial distribution The negative binomial distribution is as follows: $\displaystyle f_X(k)=\binom{k-1}{n-1}p^n(1-p)^{k-n}.$

*

*To find its mode, we want to find the $k$ with the highest probability.

*So we want to find $P(X=k-1)\leq P(X=k) \geq P(X=k+1).$
I'm getting stuck working with the following:
If $P(X=k-1)\leq P(X=k)$ then $$1 \leq \frac{P(X=k)}{P(X=k-1)}=\frac{\binom{k-1}{n-1}p^n(1-p)^{k-n}}{\binom{k-2}{n-1}p^{n}(1-p)^{k-n-1}}.$$
First of all, I'm wondering if I'm on the right track.  Also, I'm having problems simplifying the binomial terms.
 A: You are on the right track (except for typos in the notation now corrected, see @Henry's Comment and my response). Express binomial coefficients
in terms of factorials. Some factorials will cancel exactly. Others
will have factors in common: for example, 10!/9! = 10.
For a simple start, you might try the case where $p = 1/2.$
Answer: The mode is at the integer part
of $t = 1 + (n-1)/p,$ if $t$ is not an integer. For integer $t,$ there
is a 'double mode' at $t-1$ and $t.$ 
Examples: Below are three examples that illustrate this formula
(4-place accuracy):
 n = 2;  p = 1/2;  t = 3,  mode at 2 & 3
  k  :     2      3      4      5
 p(k):  0.2500 0.2500 0.1875 0.1250

 n = 2;  p = 1/3;  t = 4,  mode at 3 & 4
  k  :     2      3      4      5
 p(k):  0.1111 0.1481 0.1481 0.1317

 n = 2;  p = .4;  t = 3.5,  mode at 3
  k  :     2      3      4      5
 p(k):  0.1600 0.1920 0.1728 0.1382

Note: Wikipedia and many advanced texts use a different form of the negative
binomial distribution where only failures before the $n$th
success are counted. Hence $X$ takes values $0, 1, 2, \dots.$
For the mode according to that formulation, see Wikipedia.
(It is noted later in the article, to add $n$ to the mode
given at the head of the article for your formulation.)
