Probability of exactly $r$ books between A and B $n$ different books ($n\ge 3$) are put at random on a shelf. Among these books there is a particular book A and a particular book B. Then what is  the probability that there are exactly $r$ books between A and B . 
total case of arranging = $n!$
let $r$ books + A + B be one set then total books $(n-r-1)$
favourable case probability =$\dfrac{(n-r-1)!(r)!}{n!}$
but answer is $\dfrac{2(n-r-1)}{n(n-1)}$
 A: The subset $\{A,B\}\subset[n]$ can occupy ${n\choose2}$ different pairs of sites in the resulting order, all of them equiprobable. Exactly $n-r-1$ of such pairs have the required distance $r+1$ among the two constituents. It follows that the probability $p$ in question is given by
$$p={n-r-1\over {n\choose2}}={2(n-r-1)\over n(n-1)}\ .$$
PS: Your own thoughts about the problem, even allowing of a different way of counting, are wide off the mark.
A: If there are exactly $r$ books between books A and B, then the block of books with books A and B at the ends has length $r + 2$.  Consequently, the block must begin in one of the first $n - (r + 2) + 1 = n - r - 2 + 1 = n - r - 1$ positions.  There are $2$ ways to choose whether book A or book B is at the left end of the block.  There are $(n - 2)!$ ways to arrange the remaining books in order.  Hence, the probability that there are exactly $r$ books between books A and B is 
$$\frac{2(n - r - 1)(n - 2)!}{n!} = \frac{2(n - r - 1)(n - 2)!}{n(n - 1)(n - 2)!} = \frac{2(n - r - 1)}{n(n - 1)}$$
A: Method 1
The arrangement on the book shelf looks somewhat like
$$\text{* }\mid \text{ * }\mid\text{ *}$$
where the $\mid$ denote either books $\text{A}$ or $\text{B}$. The stars denoted on the either ends can be empty, but the star denoted in between the bars must contain exactly $r$ books. Say, the leftmost star contains $x$ books, then the right most contains $(n-r-2-x)$ books. Totals number of ways of arranging the books on the self is $n!$. The desired number of permutations is
$$
\sum_{x=0}^{n-r-2} 2! {n-2\choose r} r! {n-r-2\choose x} x!(n-r-2-x)!$$
$$
=\sum_{x=0}^{n-r-2} 2(n-2)! = 2(n-r-1)(n-2)!\\
$$
Divide this by $n!$ to obtain probability $=\frac{2(n-r-1)}{n(n-1)}.$ $\qquad \square$
Method 2
Consider the chunk of books consisting of $\text{A}$ and $\text{B}$ and the $r$ books in between them as a single object. Thus, now the number of effective objects is $(n-r-1)$. Their permutations are $(n-r-1)!$. Permutations within the chunk are $2r!$ The number of ways of choosing the $r$ books in the chunk is ${n-2\choose r}$,
hence the probability desired
$$=\frac{2{n-2\choose r} (n-r-1)!r!}{n!}$$
$$=\frac{2(n-r-1)}{n(n-1)}. \qquad \square$$
