Starting on Monday I will be teaching a (first) graduate course on the arithmetic of elliptic curves. The two texts that I will be using are Silverman's Arithmetic of Elliptic Curves and Cassels's Lectures on Elliptic Curves.
The course does not have any algebraic geometry as a prerequisite. Some students have seen a little algebraic geometry or will be taking a first course in that subject concurrently; a few have seen a lot of algebraic geometry. But at least a few have never taken and will not concurrently be taking any algebraic geometry whatsoever. One of them asked me about this, and I confirmed that the course should still be appropriate for students like him.
If you want to learn about elliptic curves beyond the undergraduate level, you will need to start engaging with some rudiments of algebraic geometry: for instance, really understanding what is going on behind the group law on an elliptic curve requires (in my opinion, at least!) a discussion of the Riemann-Roch Theorem on an elliptic curve. However, elliptic curve theory is concrete enough and the algebraic geometric input is (at the beginning) limited enough so as to make it an excellent opportunity to learn some algebraic geometry from scratch. (I think you will get a taste of that subject faster by learning some elliptic curve theory than by learning commutative algebra, although of course the latter has an essential place in the long run.)
Further, Silverman's book is especially excellently written with respect to this issue: he puts all the algebraic geometry into the first two chapters. I would -- and will! -- recommend that you begin by reading through Chapter 1 on basic algebraic geometry: it is written with a very nice, light touch and mostly serves to introduce terminology and very basic objects. Then I would skip past Chapter 2 and come back to portions of it as needed in the rest of the text. For instance, if you've never seen differentials before, I wouldn't read about them in Chapter 2 until you get to the material on invariant differentials on elliptic curves in Chapter 3.
If it freaks you out to page past two chapters on algebraic geometry, than I would recommend starting with Cassels's text. He takes a more gradual, lowbrow approach to the geometric side, but he is just as much an arithmetic geometer as Silverman, so the approach he takes is quite compatible with a more explicitly geometric perspective which may come later.
I honestly think that these two texts are so excellent that you need look no farther. If it helps, many people around here can tell you that I am very fond of writing my own lecture notes for the graduate courses I teach. However, I wouldn't dream of doing so in this case: what Cassels and Silverman have already done is essentially optimal.