It is true there is a lot of high level math in physics, some of which is done by physicists and some by mathematicians. For example the professor I am working with works with (and writes papers with) physicists very often, but the work he does is really math because it comes from a field called algebraic geometry which depends highly on pure math fields like commutative algebra and topology.
What you major in ultimately depends what you are most interested in. I am writing this from a pure math perspective, and I switched into math (from physics) because I liked the abstraction and the rigor with which everything is done. In a pure math degree almost everything you do is proofs. This is where you will take courses like number theory (or topology, group theory, analysis etc).
That being said my friends who remained in physics say that higher level physics has a lot more proofs than in the first couple of years, but they are not as abstract as the sort of proofs you would find in a mathematics course. The math courses you take with a physics degree are mostly calculus, linear algebra, differential equations. From what my friends say often the more advanced math in a physics degree is learned in physics courses (so its by nature an application) with just the introduction to the subjects in the math courses. In my university there is also an option to take a mathematical physics degree which would also allow you to take some analysis, which is more like the courses one would take in a math degree.
I unfortunately cannot say much for the kind of math you would learn in an applied math degree. The math courses you would take are similar to those you would take in a physics degree; however you would take higher level courses in these areas. From what I gather, you would learn much more of the theory behind these subject areas, which you don't necessarily get to do in a physics degree. I leave it to an applied mathematician to say more.
Despite my long answer, all in all I think the biggest differences between the math learned in math and physics, is that the math in math is very abstract, proofy, and often taught without any non-math applications. The math in physics is done always with a sense of application, but the mathematical rigor is not always taught. I would recommend taking some courses in both areas to see what interests you more.
