Using the negative binomial distr. Tom "makes"(scores) a basketball shot with the probability $0.8$. He stops when he has made 10 scores.
(A) what is the probability that he makes it in $13$ attempts
so here i used the negative binomial distribution:
$\binom{k-1}{r-1}p^rq^{k-r}$ where $k$ =  number of tries and $r$ = the number of "successes" (in our case the number of scores), $p =$ probability, $q= 1-p$ 
(B) how many attempts is it most likely that TOM needs ?
Now i could use set $p(k)= \binom{k-1}{r-1}p^rq^{k-r}$ and take the derivate w.r.t to $k$ and then find $p'(k) = 0  $ to obtain a maximum. The problem is i get a really nasty expression ..hopless. Is there another way?
 A: The probability $p(k)$ that he does it in exactly $k$ is, as you pointed out, $\binom{k-1}{9}(0.8)^{10}(0.2)^{k-10}$. 
The probability $p(k+1)$ that he does it in exactly $k+1$ is for the same reason  $\binom{k}{9}(0.8)^{10}(0.2)^{k-9}$. 
We calculate the ratio $\frac{p(k+1)}{p(k)}$. Express the binomial coefficients in terms of factorials. There will be a lot of cancellation, and after a while we get
$$\frac{p(k+1)}{p(k)}=\frac{k}{k-9}(0.2).$$
As long as this is $\gt 1$, we have that $p(k+1)\gt p(k)$, and so $k+1$ is more likely than $k$.
We get that the ratio is exactly $1$ at $k=\frac{9}{0.8}=11.25$. So for $k\le 11$, we have $p(k+1)\gt p(k)$, and after that there is decrease. It follows that the most likely number of throws is $12$.
Remark: We could instead have computed $p(10)$, $p(11)$, and so on, until the answer popped out. Instead we used a general procedure. It was done for your particular $r$ and $p$ only for the sake of concreteness. For some rather special combinations of $r$ and $k$, there will be two consecutive numbers at which the probability reaches a maximum.
