I am a physicist and I had specialized in Astrophysics back at university. Since you already have a background in differential geometry and topology, I would suggest the (one of the) two all-time classics about gravitation:
Note, however, that these are ancient books (mid 1970's). But at least at around 2000 they were still state-of-the-art in teaching. Maybe you can find more recent books. Of course, Gravitation theory as such has not changed since Einstein, but applications and probably some solution techniques probably will. And, of course, our understanding of the large scale structure of the universe has changed a lot since these times. However, dark matter and dark energy, which play a role in the latter are not at all understood, so as far as we know, gravitation theory/general relativity is still the same and only the matter that fills the universe has to be 'corrected' (presumably).
If you want to dive into supersymmetric/string theories and quantum gravitation, I cannot recommend any books because I am not into that. I would not recommend that you start with that because it would probably not give you much physical understanding (which seems to be what you're after), and it requires a lot of quantum field theoretic knowledge as well. On the other hand, you as a mathematician might enjoy the 'otherworldliness' of these theories very much. Be prepared that this has not (yet) much to do with observable physics.
If you want to understand star formation or the so-called 'big bang', there is a lot to understand from quantum field theory and especially nuclear physics. I think this is a huge topic, and you will either spend a lifetime on that alone, or you will end up reading more 'phenomenological' treatments of it.
A word of caution: physics, as it is taught, is a very different style compared to mathematics. I am seeing this from the opposite perspective than you do. While mathematicians lay out their assumptions very carefully and then prove everything, physicists often omit (presumably) self-evident assumptions and only sketch proofs. Sidenote: the most ironic thing about contemporary physics is that quantum field theory is the best working theory we ever had (predicting measurements up to 10 or so significant figures), but it still does not have a rigorous mathematical foundation. This is already painful for me, but I can hardly imagine how this must feel for a mathematician.
This is all for the purpose of getting straight to the point instead of (presumably) wasting time with trivialities. So compared to mathematics, you will need a very good memory of what was said on page 20 while you're at page 500. Most things will be referenced back, but many things will just be assumed that you remember them. And while imagination is kind of 'sugar coating' in mathematics, it is essential (with respect to fundamental physical experiments) in physics. I would even say this is rule #1: know the essential experiments/observations that led to the theories. For gravitation, this is just a handful (Basics of Electrodynamics and Michelson-Morley for special relativity, Kepler's Law, Free Fall, perihelion shift of mercury, Pound-Rebka, gravitational lensing, etc. for general relativity).
It will probably be pretty painful for you to read a physics book as it is for me to read a mathematics book. If I read a math book, I am always impatient with all the abundant notation, the lemmas and auxilliary theorems that are just building blocks for the central theorems. Physics tries to get to the point faster. You will probably find that the way theoretical physics is treated is unbearably sloppy.
Having said that, the book by Weinberg above is a little more rigorous mathematically, as far as I remember.
Of course, your mileage may vary. If you can let yourself go, theoretical physics will not be a problem for you.