Music Theory has developed tools to study music from the past. Neo-Reimann analysis had begun in the 19th century to use Euler as a basis for a more rigorous study of music. It gets complex and goes into number theory and topology. But it is rather cool and shows symmetrical relations abound in much music. If you want a start ( a rather tough one, look for books by David Lewin the music theorist). For a a simpler method that goes a different take, look at Schenkerian method which uses counterpoint to reduce works to its elements. The point of this is to learn more and more about the work as you go through the reduction. Finally, Schenkerian analysis is a useful tool in general for music theory for its offering a framework to reduce works and simplify analysis when necessary. It isn't perfect but a very valuable tool (aside from knowing the basics of music theory).
Now mathematics used in music composition - well yes algorithmic composition.The simplest being a mesuration canon. I start a melody in its original which were tones of long duration. Above that another melody is given x pitches higher and starts at time y (usually given as bar w beat z)but 2/3X faster than the original melody. Already we have some problems - what do you do when the faster version of the melody is done? Can you just repeat it without it sounding "dissonant" against the slower melody? Herein developed from practice and taste (and trial and error), rules of counterpoint - how to write two or more "different" melodies (in our example having the melody 2/3 times faster than the original ends up changing it enough to become its own melody - Ockeghem offers great examples of this) . By the time Fux wrote his book Western art music was moving away from some of the very complex counterpoint and undergoing a period of simplification. Not TOO much mind you, Hadyn amazes by how well he hides the intricacy of his counterpoint, Mozart can astound with the density and rhythmic independence of it at times and Beethoven (late works) is impressive in how florid and jagged he could make his counterpoint "work". Brahms, Wagner, Chopion, Franck deploy much contrapuntal stuff - Mendelssohn wrote some Prelude and Fugues after Bach) We don't see counterpoint regaining the complexity of the Renaissance until the 20th century. A fine example is Schoenberg's Moonfleck or Bartok's famous fugue from Music for Percussion Celeste and Strings.
Now two somewhat different melodies playing together is not something only found in Western music - homophonic, melodies at different pitches but moving mostly in tandem with each other (though the direction of each voice is not always the same) is found around the world and some brilliant stuff - Bulgarian folk music for example and folk tunes where singers may sing say a P5 and octave apart from the melody. There is heterophony which gets us closer to polyphony - everyone sings roughly the same melody but one singer may sing it plain, another embellished, another may occasionally add a few extra notes but stay close to the melody. No matter what all singers adhere to the melody. You will find this in Chinese and Indian classical music, and the rules for embellishing the melody can be quite involved yet the two lines a quite similar. Heterophony was an important step in Western art music toward polyphony, organum was when another vocalist improvised embellishments over a singer who sang a chant melody. What happened was these "improvisations" became quite complex - (listen to Hildegard De Bingen for an example) - so that they generated new melodies. Some were found unsuitable for the contemplative nature of worship and became art songs or popular songs. Soon more voices were added to the chants and the demand for more florid melodies ended up leading to the rules that eventually governed Renaissance and, later, Baroque polyphony. In addition, music notation became more complicated. So when you see sheet music for 4 voices, appreciate that it took many hundreds of years for their to be generally accepted ways to notate this music and it wasn't until the 19th century that further refinements were made to achieve what many take for granted.