The total number of sequences of 5 characters is $36^5$. Consider the sequences of 5 characters that are not valid passwords. There are two types of them: the sequences containing only digits ($10^5$ possible sequences) and the sequences containing only letters ($26^5$ possible sequences). This explains the (right) answer of the book.
Coming back to your answer, the problem is that you are counting this way the number of sequences having a digit in the first position and a letter in the second position.
Edit. If you want to use your method, you should decompose the cases as follows. Let $D$ denote a digit, $L$ a letter and $S$ any symbol. Then you may have the following mutually disjoint cases
LDSSS, LLDSS, LLLDS, LLLLD, DLSSS, DDLSS, DDDLS, DDDDL
leading to the number
26 \times 10 \times 36^3 + 26^2 \times 10 \times 36^2 + 26^3 \times 10 \times 36 + 26^4 \times 10 + 10 \times 26 \times 36^3 + 10^2 \times26 \times 36^2 + 10^3 \times 26 \times 36 + 10^4 \times 26 = 48484800
If you know automata theory, it is similar to finding the complement of a regular expression.