First thing - I don't know any algebraic geometry. I'm trying to understand a little bit about quasi-coherent sheaves but not for the sake of AG, so please rely on as little knowledge as possible.
What follows is an excerpt from Dieudonné's History of Algebraic Geometry, VIII.2.21:
"Serre's principal goal is to extend, as much as possible, to his varieties the results on sheaf cohomology described above for the classical case ($k=\mathbb C$). He restricts to coherent $\mathcal O_X$-modules (in order to be able to use the exact cohomology sequence with the definition of the cohomology groups ("Čech cohomology") of which he avails himself)..."
The exact cohomology sequence mentioned is the one induced by a short exact sequence of abelian sheaves $$0\longrightarrow \mathcal N\longrightarrow \mathcal G\longrightarrow \mathcal G/\mathcal N\longrightarrow 0$$
From this I understand the original purpose of coherent sheaves is simply the fact they form an abelian category. However, I don't understand why this justifies restricting to coherent sheaves nor why they were specifically chosen from all the abelian subcategories of $\mathcal O_X$-modules:
Couldn't Serre do his homological algebra equally well in $\mathcal O_X$-$\mathsf{Mod}$? What's the benefit of restricting to $\mathsf{Coh}(X)$? Why exactly did coherent sheaves come into play?