What are applications of number theory in physics? I was reading Goro Shimura's The Map of My Life.
He wrote the following quote in the book.
It made me come up with the title question.
In particular, is there any application of the theory of modular forms in physics?

A well known math-physicist Eugene Wigner was in our department,
  and so I occasionally talked with him. He was pompous
  and took himself very seriously. That is the impression shared by
  all those who talked with him. At a departmental party, he asked
  me what kind of mathematics I was doing. He asked that question
  as if he met me for the first time. At that time I had been a full
  professor and his colleague at least for six years, perhaps more. I
  told him vaguely, “Well, mainly things related to modular forms.”
  Then he said, “Oh, modular forms; we physicists don’t need such”
  in a very contemptuous tone.
  I can add that there are some physicists who are interested
  in modular forms. Once Edward Witten attended my graduate
  course on Siegel modular forms, and often asked sensible questions
  in class.

 A: At the 1991 AMS meetings in Orono, there was a short course offered under the title "The Unreasonable Effectiveness of Number Theory." A summary was published as volume 46 in the Society's series "Proceedings of Symposia in Applied Mathematics", LC listing QA241.U67. Three of the talks may be of interest to you: "The Unreasonable Effectiveness of Number Theroy in Physics, Communication and Music," by Manfred Schroeder; "The Reasonable and Unreasonable Effectiveness of Number Theory in Statistical Mechanics," by Georege Andrews; and "Number Theory and Dynamical Systems," by Jeffrey Lagarias. 
A: For studying applications of the number theory in physics see NUMBER THEORY IN PHYSICS by Matilde Marcolli.
Also there's a website called "Number Theory and Physics Archive" that is useful.
Finally, you can read Examples of number theory showing up in physics question on TP.SE.
A: Various kinds of modular and automorphic forms arise as partition functions (if I have things straight) in various string theories.  In more mathematical terms, certain generating functions for counting curves on various kind of surfaces or three-folds (probably Calabi--Yau varieties?) are/should be modular forms; but these generating functions have their origins in certain string theory calculations (I think with the given surface/three-fold as background).  
So modular forms are certainly important in string theory.
To give some illustrative names:
Jeff Harvey and Greg Moore are two physicists who have worked/are working on these kinds of ideas.
If you want to learn more about this, you could do worse than google these two authors names.  
A: To explain one aspect of the context of that interaction: Wigner was a very senior faculty member, had won a Nobel Prize, and was in his 60s. Shimura was a very young full professor.
Another aspect: Wigner's 1939 paper on the representation theory of (in effect) $SO(2,1)$, written to address issues of quantum mechanics, was the first substantive result on representation theory of non-abelian, non-compact groups. The 1947 paper by V. Bargmann, a physicist, was the second. No further progress on this until the 1950s when Harish-Chandra, a student of the physicist Dirac, began his systematic study of repn theory of semi-simple and reductive (Lie, and eventually, p-adic) groups.
Meanwhile, Shimura had almost single-handedly resurrected the arithmetic and algebraic-geometry aspects of holomorphic modular forms on higher rank groups, although Klingen and Hel Braun (a student of Siegel) had been ("quietly"?) working on the complex-analytic aspects, and Klingen's c. 1960 discussion of special values of (abelian) L-functions over number fields was very arithmetical. Perhaps Shimura's most special "early" contribution was to the possibility of expressing Hasse-Weil zeta functions of "Shimura curves" (as they are now known, a class generalizing "modular curves") as Mellin transforms of automorphic forms of some sort, etc.
Even the plausibility, of the Taniyama-Shimura conjecture would not be acknowledged by Weil until the mid-1960's, after his work on converse theorems. People then, and until the Wiles-Taylor work and others' in the mid-to-late 1990s, I think thought RH would be proven before Taniyama-Shimura. No one had any idea about RH, but they had even fewer [sic] ideas about Taniyama-Shimura.
Wigner would not have known about Weil's conjectures, nor the nascent algebraic geometry required to put them in any context. Shimura might not have believed that Harish-Chandra's repn theory, beginning with Wigner's result, would, as explained by Gelfand and his school, and Selberg, and taken up by Langlands et al, provide an over-arching context for not-necessarily-holomorphic automorphic forms, if not their "arithmetic".
The other "human" aspects of the situation we can imagine easily...
But, even beyond the human-foible aspect, it is absolutely not surprising that Shimura was not in awe of Wigner, and that Wigner had no reason to care much about Shimura's work.
Witten's interest was quite a few years later, after Shimura's, Selberg's, Harish-Chandra's, Langlands', and many others' work had made clear that the special objects studied in "number theory" strongly resembled the special objects of parts of physics. Not to mention that Witten is more of a "visionary" than many of us. And won a Fields Medal, so maybe he's a good mathematician, too? :)
From my personal viewpoint, apart from those historical observations, I note that the specific mathematics on arXiv that seems relevant to my concerns, second after "number theory", is the "math-ph" section.
As an example, the van Hove (et al) differential equations that (I hear...) model something about graviton interactions, are precisely the same genre of differential equations "in automorphic forms" that appear in various spectral-theoretic scenarios, going back to Anton Good's papers in the early 1980's, and continuing in various peoples' work today. Steven Miller at Rutgers, a guy who "does" automorphic forms, has actively collaborated with that physics groups, for example.
Indeed, Rudnick, Ueberschar, Marklof, and their collaborators often say that they are doing "mathematical physics", and are in "physics institutes", ... but their work looks to me like a study of number-theoretic aspects of harmonic analysis... which would extend to be "automorphic forms", if taken on to more difficult cases.
And, finally, probably autobiographies do not reliably involve scholarly reconsideration of much of anything at all, as they are reminiscences... so scientific accuracy is by far not guaranteed.
