I'm a non-mathematician who is self-studying mathematics. Although I'm very interested in mathematics, my main purpose is to apply math in theoretical physics. The problem is that when I read a mathematics books, I can't see a clear way to apply these math in a concrete setting. I want to apply higher math in my study of theoretical physics (not mathematical physics). I'm not looking to put physics on a rigourous basis (e.g axiomatic field theory). I want to use math (e.g. category theory and algebraic geometry) in order to discover new ways of thinking about physics, generalizing concepts and to calculate stuff. I'm completely self taught in math. Should I read pure mathematics textbooks aimed at mathematicians? What's your advice on this?
While studying physics as a graduate student, I took a course at the University of Waterloo by Achim Kempf titled something like Advanced Mathematics for Quantum Physics. It was an extraordinary introduction to pure mathematics for physicists. For example, in that course we showed that by taking the Poisson bracket (used in Hamiltonian mechanics) and enforcing a specific type of non-commutativity on the elements, one will get Quantum Mechanics. This was Paul Dirac's discovery. After taking his course I left physics and went into graduate school in pure mathematics.
(I don't believe he published a book or lecture notes, unfortunately, though I just emailed him.)
In transitioning from physics to mathematics, I learned that the approach to mathematics is different in a pure setting than in a physics setting. Mathematicians define and prove everything. Nothing is left unsaid or stated. There is an incredible amount of clarity. Even in theoretical physics, I found there to be a lot of hand-waving and ill-defined statements and lack of rigor (which hilariously caused me a lot of anxiety). Overall, though, Mathematicians are focused on understanding and proving relationships between abstractions, whereas physicists are more interested in using these abstractions as tools. Therefore, the approach is very different: mathematicians don't care what the application is, they only want to understand the object under consideration.
Nevertheless, for a theoretical physicist looking to get a firm background in mathematics, you want to have the following core mathematical concepts, which will provide a foundation to explore any avenue:
But the real list is something like:
Set, Group, and Ring theory are used extensively in physics, especially in Hamiltonian mechanics (see Poisson Bracket). Real Analysis and Linear Algebra are needed as a foundation for Functional Analysis. Functional Analysis could be described as an extension or marriage of Ring Theory, Group Theory, Linear Algebra, and Functional Analysis. Therefore, many concepts in functional analysis are extended or used directly from Real Analysis and Linear Algebra. Measure Theory is important for the theory of integration, which is used extensively in applied physics and mathematics, probability theory (used in quantum mechanics), condensed matter physics, statistical physics, etc.
Topology and Operator Algebras are used extensively in advanced quantum mechanics and Relativity. Specifically, Algebraic Geometry is studied extensively in String Theory, whereas Topology is used extensively in General Relativity. Operator Algebras are an important area for understanding advanced Quantum Mechanics (ever heard someone talk about a Lie Group before?)
Some canonical text-books I would recommend:
Those are some decent text-books. I would say: give yourself two years to digest that material. Don't be hasty. Remember: mathematics is about definitions and proofs. Do not expect to see "applications" in any of those books. Just understand that the concepts are needed in advanced physics.
Unfortunately, though, I don't know of any text-book that forms a direct bridge between the two. If Achim Kempf had published his lecture notes, those may have worked, as essentially, he was doing just that.
There are some nice books that can help bridge the gap between mathematics and theoretical physics, though unfortunately not enough.
One book that I would highly recommend is V.I. Arnold's "Mathematical Methods of Classical Mechanics", A text that gives the reader quite a lot of intuition onto the meaning of the mathematical concepts of manifolds, lie algebras, etc. Although the book discusses Classical Mechanics, I think this is a good starting point even for more advanced topics such as QFT.
Pure math books may not be the best thing for you. Your job as a theoretical physicist is to make-up the math you need. Allowed operations in calculations follow from physical more than mathematical principles. Once the dust settles, then you look around and see if mathematicians have developed tools which fit your calculus.
That said, I recommend the text explaining supersymmetry for mathematicians by Varadarajan. The pdfs posted at the author's website below contain a fair amount of the text.
(It has much more than just supersymmetry)
Beyond that, there are many good books. However, what you really need to do is pour over recent papers on the ArXiV.
First of all I strongly recommend a great two-volume book by R.Courant and D.Hilbert Methods of Mathematical Physics, vol.1 (linear algebra, series expansions, integral equations, calculus of variations). vol.2 (partial differential equations) - the best ever written book on mathematical methods of physics, uncomparable to the most of "modern" books on the subject. It comprises the (broad) base for further interests. Although these volumes are quite old they are still crucial if you want to apply mathematics in theoretical physics.
To point out more recent books let's mention (a bit over 30-years old but many times edited and revised ) great book by three women Yvonne Choquet-Bruhat, Cécile DeWitt-Morette, Margaret Dillard-Bleick : Analysis, Manifolds, and Physics. There is also the second volume. Y.Choquet-Bruhat wrote (2009) also General Relativity and the Einstein Equations. For over 50 years she contibuted to mathematical analysis of Einstein theory, especially to Cauchy problem.
A very well written (3-volume) book mainly on geometry : Modern Geometry--methods and Applications by B.Dubrovin, A.Fomenko, S.P. Novikov (the Fields prize 1970). The book covers main issues of geometry (tensors, manifolds, bundles, calculus of variations) with many references to physics ( Yang-Mills, Einstein equations etc.).
A very interesting book deeply discussing issues concerning calculus of variations is The Action Principle and Partial Differential Equations by Demetrios Christodoulou. It may be hard to read but it is really rewarding. It may open your eyes on the variational principles and their fundamental importance in modern theoretical physics.
These are the first words in a beautiful article On teaching mathematics by a preeminent mathematican V.I. Arnold : Mathematics is a part of physics. Physics is an experimental science, a part of natural science. Mathematics is the part of physics where experiments are cheap. The Jacobi identity (which forces the heights of a triangle to cross at one point) is an experimental fact in the same way as that the Earth is round (that is, homeomorphic to a ball). But it can be discovered with less expense.
No matter if you agree with him or not , if he is right or not. These words say by themselves on great importance of intimate relations between physics and mathematics. It is very advisable to read Arnold's another article on the subject Mathematics and physics: mother and daughter or sisters?
I wouldn't like to distinguish mathematical and theoretical physics, there is rather a good or bad theoretical physics. Well, we would say that Einstein, Maxwell, Boltzmann, Dirac were theoretical physicists, while Hilbert, Poincare, Minkowski were mathematicians, nevertheless all they had their very important impact on the modern physics. Nowadays theoretical physics seems to be close to purely mathematical concepts, being rather far from experimental evidence. Unfortunately it may be still far from The Road to Reality. By the way Roger Penrose started his scientific career in algebraic geometry and next he made a great contribution to mathematical physics, especially to general relativity.
It would be difficult to distinguish mathematics and theoretical physics talking about General Relativity, especially there is so called Mathematical Relativity concerning exact results on analysis of Einstein equations. One would say that people in mathematical general relativity are mathematicians working in a physically interesting theory, and it's true but on the other hand they do really interesting things also for experimental physicists. One of the main contributor to Mathematical Relativity S. Klainerman wrote a great article PDE as a Unified Subject where discussing the main issues of the research in partial differential equations he included many interesting remarks on relations between mathematics and physics. There are numerous references to the subject.
To conclude I would refer to one of the greatest mathematicians of XX century - Hermann Weyl (D.Hilbert's student). At the dawn of the Einstein theory of gravity, he wrote one of the best books on the subject (not very technical) Space Time Matter.
All those books and articles are recommended and certainly helpful in a fascinating world of theoretical physics and its deep connections to mathematics.
It may be interesting to you to look at the math overflow question here: http://mathoverflow.net/questions/71909/book-on-mathematical-rigorous-string-theory/71919#71919
The answer I gave there points to a set of lecture notes that discuss a rigorous treatment of quantum field theory from the mathematical point of view. Edward Witten IS a physicist and mathematician, so his notes are very nice to read. Effort spent thinking about the basic ideas presented here and their physical meaning is, in my opinion, not wasted.
Witten's notes on Field Theory can be found here: http://www.math.ias.edu/QFT/spring/index.html
The rest of the notes also are available for free (there is a link to the notes in a comment in the above MO answer).
One book which in my opinion is very good to connect mathematical concepts with quantum mechanics is Geometry of Quantum States by Ingemar Bengtsson and Karol Życzkowski. It is not a mathematical textbook (so if you want to learn the concepts with mathematical strength, you'll have to read in addition mathematical books about those concepts), but it definitely tells you about the connection of those concepts with physics (more exactly, with quantum states, as the title already says, and you'll be surprised how much mathematical structure you find in quantum states alone).
Mathematics for engineering books can be useful for real examples of mathematical application in real world situations. Had you thought about that, or is that a bit too practical for your purposes?
I was recommend "a guided tour of mathematical methods for the physical sciences" by Roel Snieder. It seems like a good book to use if you are not following a specific course but want a more complete understanding of what you are doing with maths in physics. It has examples at each point rather than at the end of the chapter, which I prefer because I usually forget if I don't try it at once, myself.
I have also heard good things about Mary Boas' book: Mathematical Methods in the Physical Sciences, but have no personal experience other than that.