For a more computational introduction and definitely manageable to work through on your own, I would suggest one of "A Friendly Introduction to Number Theory" or "Elementary Number Theory by Jones and Jones". I prefer the former, as it get's to some pretty complex stuff (elliptic curves by the end I think). The Jones and Jones is good too though, and manages to sneak a little introductory algebra without you noticing.
If you're looking for more of light read with fewer (maybe none?) exercises, "Higher Arithmetic" is good and helpful for background and intuition for some of the intro concepts.
Note that none of the above assume really any machinery from college math, just some maturity.