Why did Serre choose coherent sheaves? First thing - I don't know any algebraic geometry. I'm trying to understand a little bit about quasi-coherent sheaves but not for the sake of AG, so please rely on as little knowledge as possible.
What follows is an excerpt from Dieudonné's History of Algebraic Geometry, VIII.2.21:

"Serre's principal goal is to extend, as much as possible, to his varieties the results on sheaf cohomology described above for the classical case ($k=\mathbb C$). He restricts to coherent $\mathcal O_X$-modules (in order to be able to use the exact cohomology sequence with the definition of the cohomology groups ("Čech cohomology") of which he avails himself)..."

The exact cohomology sequence mentioned is the one induced by a short exact sequence of abelian sheaves
$$0\longrightarrow \mathcal N\longrightarrow \mathcal G\longrightarrow \mathcal G/\mathcal N\longrightarrow 0$$

From this I understand the original purpose of coherent sheaves is simply the fact they form an abelian category. However, I don't understand why this justifies restricting to coherent sheaves nor why they were specifically chosen from all the abelian subcategories of $\mathcal O_X$-modules:

Couldn't Serre do his homological algebra equally well in $\mathcal
 O_X$-$\mathsf{Mod}$? What's the benefit of restricting to
  $\mathsf{Coh}(X)$?
  Why exactly did coherent sheaves come into play?

 A: All Serre needed to use was quasicoherent sheaves. For these it's a fundamental theorem that Cech cohomology is the "correct" cohomology. That means that it agrees with the cohomology defined abstractly, via derived functors. Since derived functors give rise to long exact sequences, the fundamental theorem implies that short exact sequences of (quasi)coherent sheaves yield long exact sequences in Cech cohomology. This just isn't true for coarser abelian categories like arbitrary sheaves or presheaves. 
For an idea of why the theorem is true for quasicoherent sheaves, it's generally true that Cech cohomology becomes the right cohomology when the space has a "good cover," which in this case means a cover whose elements have no higher cohomology for whatever sheaf we're investigating. This is always true for quasicoherent sheaves by a reduction to affine varieties and then commutative algebra, which isn't possible for general sheaves since there may be no cover on which the sheaf is that associated to a module over a ring.
A: Serre didn't mention  quasi-coherent sheaves nor derived functor cohomology  (=Grothendieck Tohoku cohomology, introduced in 1957) when he finished writing  FAC in October 1954, because nobody in the world knew about these at that date.
He didn't know about abelian categories either for the same reason.
He introduced coherent sheaves in algebraic geometry in imitation of the notion of coherent sheaves in analytic geometry, where they had been created by Oka and formalized by Henri Cartan and Serre himself.
In particular one of the triumphs of FAC was the proof in the algebraic geometry context of the Cartan-Serre theorems A and B.
The only sheaf cohomology used by Serre  was Čech cohomology.
The demonstration that one could  use the absurdly coarse Zariski topology to obtain powerful results for algebraic varieties stunned algebraic geometers, foremost among them Zariski (!) and Grothendieck, who was initiated to algebraic geometry through FAC. 
