I often wonder the same thing, and for concerns and interests around these lines found myself reading "The Cambridge Handbook of Expertise and Expert Performance." (I would suggest reading this more if you are interested - it is a well-supported book and says lots of things like practicing well for a few decades along with a supportive home environment is vital.) To put your concerns to rest, I was going to quote the aforementioned tome, but I couldn't find the quote (it's 800 pages) I was looking for and thus can only paraphrase: they found little to no correlation between intelligence levels (as measured by IQ tests) and internationally recognized performance (nominated by peers) amongst people with PhD's. Put another way, amongst mathematicians, physicists, molecular biologists, etc. those with an ostensibly higher level of intelligence have little to no greater chance of being amongst the elite few who are nominated by their peers for/win international level awards.
Hence I would say that if you have the potential to earn a PhD and begin research in mathematics, then it is likely that other qualities will determine your success; and thus hard work will indeed influence your success greatly. Note that this view doesn't necessarily fly in the face of the fact that some people are indeed smarter than others; it simply acknowledges that having a slightly higher IQ relative to others with very high IQs is far less of an advantage than other, more "soft" qualities and of course luck. Terry Tao says:
Does one have to be a genius to do mathematics? The answer is an emphatic NO. In order to make good and useful contributions to mathematics, one does need to work hard, learn one’s field well, learn other fields and tools, ask questions, talk to other mathematicians, and think about the “big picture”. And yes, a reasonable amount of intelligence, patience, and maturity is also required. But one does not need some sort of magic “genius gene” that spontaneously generates ex nihilo deep insights, unexpected solutions to problems, or other supernatural abilities... Of course, even if one dismisses the notion of genius, it is still the case that at any given point in time, some mathematicians are faster, more experienced, more knowledgeable, more efficient, more careful, or more creative than others. This does not imply, though, that only the “best” mathematicians should do mathematics; this is the common error of mistaking absolute advantage for comparative advantage.
This all fits in with one's sort of intuitive picture of things (if one has read biographies and such): certainly Einstein was extremely clever, and quite gifted at mathematics, but when comparing his first-blush mental abilities him to someone like von Neumann, one thinks Einstein was quite dumb!!! Yet both contributed seminal ideas to their fields...