Complementing points made in other answers, and Akhil M's question/comment: it's not only the prestige of the highest-ranked grad programs, but the arguable fact that one is exposed to "better" mathematics there. E.g., the chances that one is guided to an "interesting thesis" are probably higher at higher-ranked schools. This presumes that one pays some attention to the advice one is given by the presumably-first-rate faculty there, and assimilates presumably-first-rate attitudes and viewpoints about methodology, goals, and taste. I think ideally grad school changes a person to be a better mathematician. It's not just dues-paying and hoop-jumping, while waiting to write up one's clever ideas. Certainly if one is not open to learning new things (except superficially), it can be degraded to that, but it oughtn't be.
(Edit: it is plausible that the internet allows access to previously inaccessible things, but the internet includes "everything", and, while judgemental in some senses, is not critical... Immediate personal contact with people may yet have some purpose...)
As Pete Clark and others did/would comment, it does appear to be the case that the ranks of grad school, postdoc, and tenured position are non-increasing, probably decreasing significantly. Not only is this statistically observable, but there are some not-crazy reasons for this dynamic. The most obvious is just the counting argument about how many PhD's are produced, versus hiring. That alone might swamp all other considerations.
About GRE subject test scores: by all means retake the thing if you had a disappointing number. Even though people realize that it's a fairly silly test, there seem to be enough talented students out there who can cope with this silly test so that it is feasible to use it as first-pass filter.
The prestige of your letter writers is very important, and what they say is very important (despite the inflation).
The flip side of some of this is the self-referential nature of "the best mathematics/program": the best math is what is done at the best schools? :)
Similarly, do you want to have a successful research career in your own perception, or in someone else's? These are not reliably the same thing. One could argue that the strictures of "top-level research" are fairly tight... and this might not be exactly the mode of operation you want.
Finally, I think it depends greatly on one's personality as to whether the most challenging situation possible is best, or, rather, a more forgiving, indulgent situation. Mixed in with this is the clumsy or accidental stress created in many human interactions, and the statistically typical communication difficulties of people in mathematics. That is, depending on the person, and depending on the details, stress can be productive or destructive. Oddly, the GRE and such things are one sort of "stress test", although I am not confident that they measure very much about mathematics. Is it necessary to be "tough" to be a mathematician? No, certainly not, but it may have as corollary that one can cope with more circumstances than others...? While this surely is a general advantage, still, the details matter.
In summary, trying the GRE again is reasonable, and getting the best letters you can. Apply to a range of schools (even though it costs money: the investment is worth it). Grad schools really don't want to admit anyone they don't feel will succeed, and they're very practiced at gauging these things, so there's little risk that you'll accidentally find yourself out of your depth, even if you are not sure yourself at this moment what the depth is. That is, get your application looking the best you can manage, and let the grad programs appraise you.