What is the mathematical name of the "curved space" in Einstein's theory? Newtonian mechanics assumed a Euclidian geometry. 
Einsteinian mechanics is based on non-Euclidian geometry. 
However, there are many types of non-Euclidian geometry, so what is the mathematical name of the geometry used in Einstein's relativity theory?
 A: Riemannian geometry is the backbone of Einstein's General Theory of Relativity.
To quote, in his own words:

"[In 1912] I suddenly realized that Gauss's theory of surfaces holds
  the key for unlocking this mystery. I realized that Gauss's surface
  coordinates had a profound significance. However, I did not know at
  that time that Riemann had studied the foundations of geometry in an
  even more profound way. I suddenly remembered that Gauss's theory was
  contained in the geometry course given by Geiser when I was a
  student... I realized that the foundations of geometry have physical
  significance. My dear friend the mathematician Grossmann was there
  when I returned from Prague to Zürich. From him I learned for the
  first time about Ricci and later about Riemann. So I asked my friend
  whether my problem could be solved by Riemann's theory , namely, whether the invariants of the line element could
  completely determine the quantities I had been looking for."

Albert Einstein, as quoted by Abraham Pais in Subtle is the Lord,
 Pais's scientific biography of Einstein.
