Where can I learn more about the implications, meta discussions, history and the foundations of mathematics?
Is Russell's Introduction to Mathematical Philosophy a good start?
Philosophy of Mathematics: Selected Readings, edited by Paul Benacerraf and Hilary Putnam, is one of the standard essay collections, and introduces the classical schools: formalism, intuitionism and logicism.
But there are newer points of view. Personally, I liked The Nature of Mathematical Knowledge by Philip Kitcher because of its sophisticated historical point of view. Kitcher's approach is empiricist.
Another interesting approach is Charles Chihara's structuralism, as presented in his book Constructibility and Mathematical Existence.
This next one is not a book but it is famous enough that I think it should be mentioned here. With respect to axiomatic set theory and philosophy, there is the two-part essay entitled "Believing the Axioms" by Penelope Maddy:
Maddy, Penelope (Jun. 1988). "Believing the Axioms, I". Journal of Symbolic Logic 53 (2): 481–511.
Maddy, Penelope (Sept. 1988). "Believing the Axioms, II". Journal of Symbolic Logic 53 (3): 736–764.
If you are interested in the foundations of set theory in particular, there is the classic book Foundations of Set Theory by Abraham Fraenkel, Y. Bar-Hillel and A. Levy. The standard system of axiomatic set theory ZF is named after Ernst Zermelo and Abraham Fraenkel.
I think there are probably some good introduction to "classical" philosophy of math that I'm not aware of, but what I find most interesting are modern treatments of philosophy of math.
Lakoff and Nunez's book called Where Mathematics Comes From: How the Embodied Mind Brings Mathematics into Being uses modern cognitive science to discuss philosophy of math.
David Corfield's Toward a Philosophy of Real Mathematics is a fascinating look at what mathematicians actually do (as opposed to most philosophy of math done by philosophers that don't know much math).
I have lots of other suggestions, but I'll let other people have their say as well (and moreover this wasn't quite what you were asking for).
Russell's book is probably not what you're looking for. Firstly, it is mostly his opinions on the subject, and some of his arguments are surprisingly weak (as I see them anyways). Here is the online version of Russels book. Looking at the content list, it does not seem what you are looking for.
Personally I can recommend "The Mathematical Experience", by Davis and Hersh. http://www.amazon.com/Mathematical-Experience-Phillip-J-Davis/dp/0395929687 It presents some of the main philosophical views on mathematics. (among other things)
When I took a course on the philosophy of mathematics, our textbooks were Shapiro's Thinking about mathematics and Parsons' Mathematical Thought and its Objects. I recommend them both.
Not a book, but a very scholarly overview article on the Philosophy of Mathematics from the Stanford Encyclopedia of Philosophy:
Since no one mentioned it,
Other's have suggested very good books for the history and the foundations part of your question, so I will just mention one which I think is a great fun read and addresses some of the "meta discussions" you asked about: Gian-Carlo Rota's Indiscrete Thoughts. Rota is, to me at least, quite interesting as his book not only gives lots of nice bits of Math history and anecdotes about all of these mathematicians that you might otherwise only think about as names attached to theorems, but also because his position as a philosopher and mathematician makes him talk about ideas related to math that I have never seen discussed elsewhere (the chapters "the pernicious influence of mathematics on philosophy" and "the pernicious influence of philosophy on mathematics" are good examples of this).
Also, of course, Rota is known as quite a character and spends a great deal of this book living up to that--taking joy in arguing against ideas in math and philosophy he doesn't agree with, and in telling slightly off color stories about Ulam and von Neumann, which does make the book a great deal of fun.
"What is Mathematics, Really?" by Hersh, though the main thrust of his own views is social-constructivism, gives a great summary of the main schools of thought (including the classics of platonism, logicism, formalism and intuitionism). (of course, a lot of philosophy of mathematics is not covered by a 'school')
If you’re looking for a convenient brash-up of mathematics, I genuinely would recommend Mathematics: A very short introduction by Timothy Gowers. The book does not require hard practiced mathematics and the author rans a philological and a mathematical approach towards the subject. Plus, the author Timothy Gowers, is the Royal Society Research Professor at the Department of Pure Mathematics at the University of Cambridge. The chapter in abstraction is a splendid joy to read.
Morris Kline : Mathematical Thought from Ancient to Modern Times, three volumes
Ian Hacking's book is a must:
Hacking, Ian. Why is there philosophy of mathematics at all? Cambridge University Press, Cambridge, 2014.
What I find particularly illuminating is his distinction between separate approaches to historical interpretation of the evolution of mathematics: the butterfly model (implying inner necessity as the evolution of a biological organism e.g., a butterfly) and the Latin model (as in the evolution of a natural language, implying a degree of contingency depending on social and cultural factors).