What are the mathematical foundations on which numerical modeling and simuations are based? I.e. which topics should one study in order to be able to do research in these topics, coming from a physics background. A recommendation of good ressources for self-study would be highly appreciated.
As @Henry has commented, there is a big difference between the background needed to do research in mathematics and simulation and the background needed to use such methods for research in other fields.
Many numerical simulations are based on having a good source of pseudorandom numbers. Traditionally, a lot of research in that area has involved number theory and computer science. But now we have some excellent random number generators for most purposes (perhaps with the exception of encryption), and I'm not sure the direction research for making future generators might take.
Both probability methods and differential equations have been used to simulate physical systems, and some simulations require background in CS. If visual displays of simulated results is involved, then some linear algebra would help.
Eventually, for success you will need to settle on a match between the kinds of research that interest you and the additional mathematics and CS you feel comfortable learning. There is probably no substitute for talking to some people who are doing the kinds of things you'd like to do, letting them know something about your background, and getting their advice.