First of all, you say
D a divisor (that can be not effective) such that (H.D)<0
There is a bit of ambiguity in your language, so let me just say to be clear that such a $D$ definitely cannot be effective. This is because a hyperplane section of a surface (in any projective embedding) is an ample divisor, so it has nonnegative intersection with any effective divisor (and in fact positive intersection with any nonzero effective divisor).
It then follows that the linear system $|D|$ must be empty, simply because intersection numbers are preserved by linear equivalence: if $D'$ was an effective divisor such that $D' \sim D$, then we would have $H \cdot D' = H \cdot D <0$, which is impossible.
One good source for divisors on surfaces without scheme language is Chapter 4 of Shafarevich, Basic Algebraic Geometry Volume 1.