As @Mike Spivey commented, there are wildly different trajectories/situations, and depending which of these is acceptable or desirable to you, the answer to your question is wildly different. There are (at least) "research" jobs, "liberal arts college" jobs, and "branch campus jobs", and, as you yourself mention, there is also the "low-caste" job of "adjunct", which exists at "research" universities and "branch campuses" both. The "endless postdoc" possibility is not actual these days, I think. If you do not understand the distinctions between these, it might be wise to look into it. Of course this kind of answer is U.S.-oriented, which seems to be the questioner's orientation.
The top-or-second-tier research university job is very cool if you can get it, can endure the stress, and can get tenure. Also, you have virtually no idea of where you'll be living, since you apply all over the U.S. (and/or Canada). This does raise the issue of "research university out in the middle of nowhere" versus "research university in a metropolitan area"... To get either one of these sorts of jobs is like managing to get a job in a major symphony orchestra: good pedigree helps, sufficient "minor virtuosity" is necessary, solid technical chops, and well-known people who've heard of you and are willing to endorse you. Apart from the quasi-objective aspects, there's also the "are you connected?" aspect, which easily drifts into politicking and such. (As do many human activities, indeed, so it's not special to this situation... but/and things are not "objective", is the point.)
Trendiness of what you work on has a huge impact, as does the place you do your graduate work, the reputation of your advisor, and what your advisor says about you.
Being a decent human being, or being a good teacher, or anything else doesn't matter terribly much in the "research" game, unfortunately, though there seems to be slight improvement in comparison to 1970-80. In fact, sadly, "kindness is often mistaken for weakness", etc. Too bad, though!
On the other hand, if one is for some reason unavoidably drawn to mathematics, and is really quite good at something in it (yes, there's the issue of figuring out what you're good at, but don't decide based on grades in graduate courses or anything so superficial...), the stresses are probably comparable to any sort of high-flier executive position... and, unlike the latter, at a certain point one does have the luxury of "stopping to think" about one's own goals. Not for a while! But at least the time does come. Few other jobs offer that, no matter what the initiation fee.
The four-year school job game I have no real idea about. The "adjunct at main campus" I see every day, and there are profound ideological problems there. Often, adjuncts are not given pictures on the photo gallery in the hall, are not acknowledged as any sort of "faculty" at all, despite having taught low-level classes for decades. Pay is lousy. The AAUP militates against mis-treatment of adjuncts, but the economic downturn increases vulnerabilities...
There are relatively very many "branch campuses" of state universities, where pre-calculus and calculus are the staples, and student-friendliness and being-a-good-citizen are paramount. Teaching loads are quadruple the "research university" loads, but there's much less pretense, less craziness, and less craziness tolerated as though it were excused by possession of esoteric powers. A different world. Several of my students, capable as they were at "research", have found "little college" life far more convivial than "research..." tracks.
(A recent tendency of small colleges to decide that their "REU" programs are real research, and, therefore, that their business is "research", rather than teaching 20-year-olds basic things, is horrible. Apart from being sadly delusional, it disserves all parties.)
I'm sure other people will have other viewpoints.