In my opinion, mathematics is a very poor fit for the standard university model of lectures, presented linearly, over the course of a quarter/semester.
If you fall a little behind in a math course, it is painful, if not impossible, to catch up. This is because it is not enough to learn the material—you have to practice it, and in doing so you might discover very deep issues in your understanding of the background ideas. Yet the course speeds along, giving you no time at all to polish what needs polishing.
We, the teachers, handle such situations very badly. We tell ourselves that these students weren't ready for the course, that they should have taken something else first, or, worst of all, we think that they aren't "true mathematicians". But this is all wrong. Becoming great at mathematics involves all sorts of false starts and reworking of basic techniques. The people who are "ready" for the course are often the people who get the least out of it; they could as well have read the book!
Math should be taught in a way that allows students to explore their weaknesses more deeply. This is impossible to do if every student must be learning the same material at the same time.
I was lucky enough to have access to an at-my-own-pace style of math education throughout high school. In college, I felt like I was a great student, always on top of my math classes. Looking back, the classes were far too slow for where I was, and I was just a terrible student with the right background. I wasn't being challenged, and so I learned a lot of bad habits, which made grad school very, very painful for me until I remembered how to push myself and focus on my weaknesses again.
So I don't think that math goes too fast exactly. I don't think that math should have any pace at all, but be driven by the needs and abilities of the student. This is not an easy ideal to achieve, of course, but I think that everyone in mathematics too easily accepts the myths surrounding the system that now exists. Those at the top become arrogant, and everyone else becomes anxious and fearful. You are right. We can do much, much better.