I completed my PhD in pure math (differential/Riemannian geometry) a few years ago, and have recently turned my attention to quantum field theory and string theory, purely out of interest and curiosity.
I've learnt that physicists seem to use a different style of logical reasoning and derivations in their work, and a different process in the development of their body of knowledge. They largely use the physics of a problem to guide their derivations and reasoning. They use the scientific process of hypothesis and experiment to guide the development of physics - what passes as acceptable knowledge in physics is ultimately that which is validated by experiment to an acceptable degree of accuracy. Physicists are interested in the final result of their reasoning of explaining the universe around us to an acceptable degree of precision, not necessarily in the level of mathematical rigour they displayed in getting to this result. They do physics not mathematics. These things take some getting used to and I understand that it couldn't really be any other way.
However, what frustrates me is I've noticed that physicists often use their own mathematical notations, definitions, terminology and even mathematical reasoning, even when clearer, simpler and in general better ways of doing things have been developed and highly refined within the mathematics community over a significant period of time. Physicists also often seem to assume large amounts of universal symbols, notation and terminology - they often proceed with doing a significant amount of work, derivations, reasoning, etc, with undefined terms and mathematical notation, assuming that the reader already knows the notation and terms as though they are unambiguously and universally accepted in the wider physics community and set in stone. I've read some theoretical physics books where the authors have used, and significantly drawn upon throughout the whole book, certain mathematical symbols and notations which they never actually defined in the first place.
You asked the question: "but I was wondering if more advanced physics/mathematical physics is as rigorous as pure math? Or is this lack of rigor something I will just have to accept as I move on in physics?"
My experience is that mathematical rigour is less taken into consideration as you get into more advanced mathematical/theoretical physics. Things get very confusing and if you don't understand or are unable to verify something you're reading, it becomes very difficult to discern if it's due to just not following along or that what you're trying to understand or verify actually isn't well founded mathematically.