I've always loved playing with numbers, but haven't had any formal guidance in the study of advanced mathematics and number theory. Is there a book (or a few books) on mathematics that I wouldn't have learned in my high school curriculum (algebra, trig, and calculus) that could help me understand higher level maths and get me started on number theory? I know it's quite a broad question, but when I see things like this xkcd, and in trying to understand concepts I go to Wikipedia articles to find them chock full of terminology that flies over my head, it's quite frustrating.
That xkcd strip is not about number theory, but about set theory (and a particularly slippery part of set theory at that; the Axiom of Choice is tricky!).
A possible first set theory book might be Halmos's Naive Set Theory, though it can be tough in parts and suffers somewhat for having few exercises.
There are several introductory Number Theory books; e.g., William Stein's Elementary Number Theory: Primes, Congruences and Secrets; Leveque's Elementary Number Theory; and Underwood Dudley's book of the same title.