I will take your question in reverse.
We all wish we had enough time. I certainly wish that I could live to see the Starship Enterprise, and I would be happy to mop the floors there if I could be so fortunate.
Yes (emphatically) you will be doomed to a life of constantly being behind those who are smarter than you. You really need to accept this because it is reality. You may be suffering from a very common thought paradigm. Many people, especially when they are younger, base their own self worth on their ranking among their peers. The truth of the matter is that no matter how good, talented, or gifted you are, there is probably (degree of probability very near unity) someone that is much better than you.
Stop competing with your peers. Take neither pleasure nor pain in how your performance compares to your known peers. Shut the thought of comparing your own performance with that of your known peers out of your mind forever, and take neither joy nor sadness in these sorts of results in the future. You need a real peer to compare your self worth and performance against. I can tell you from very deep experience that this peer is you. You can spend the rest of your life trying to outdo yourself, and you will actually succeed sometimes. After that success you will still have your biggest challenge in the mirror. If you want true joy out of success in life, then push your own envelope of ability, and when you "outdo" yourself, take pleasure there. You really are and will always be your own greatest competitor, take my sage advice here.
I have a secret for you. I will share it here on stackexchange, but as it turns out, most people will not actually take advantage. By definition, most people are mediocre. This applies to the smartest people ever born. Einstein was probably to the right of 4 standard deviations in antimediocrity, who was known as a genius (that sucked at math). He actually was bright, but his true gift was that he shot for the stars. You can take a lesson from what I say, and take comfort in the fact that most people, even the brightest, are mediocre in life. All of the natural born talent in the world rarely results in an Einstein. The bigger factor usually breaks down to a willingness to do the hard work. You can top the brightest of minds just by shooting for the stars in your own effort. Attempt to outdo yourself and only yourself daily.
GRE tests are crap. Shut that whole deal out of your mind. Most schools do not take them seriously, they just require them.
When you graduate, you will be ready for graduate school. By virtue of your diploma you have actually prepared for graduate school. Shut this malarkey about other people being better, or the things that you do not know out of your mind.
That being said, you should do a few things to prepare, and you should start doing them tonight and for the rest of your life.
Do not be a mediocre slackard like the average person. When given a choice, take the harder class and not floating in the swimming pool 101.
When you are taking a math class (or any class) do not wait for a teacher to tell you what to learn and when. Come up with your own questions, and enjoy the struggle of discovery. Grab knowledge by the cohojones!
Do not be afraid of failure. Do be afraid of not doing your best.
You definitely need to do well in your classes when you are taking classes. After those classes, if you really want to get strong in math then get yourself into the habit of doing about 6 hours per day of just math, and plan to continue this for several years (until you decide to retire). Review your studies, explore new ideas, read, write, solve problems, and attempt the proofs. Euler did it until the day he died. We call him a genius, and I am absolutely convinced that Euler was one of the greatest mathematical minds in all of history, but I suspect that much of his success came from the fact that he spent all of his days doing the math with a spirited unquenchable passion for all things math that continued even through blindness and ended on a gentle summer day when he probably died in the midst of a great problem in applied mathematics.
Physics and computer science are actually ideal, probably optimal skills for a degree at any level in applied mathematics. Capitalize on this. Programming skills are especially important, and you should not let them go away. You will definitely need them, so do not discount them. Computer science is really a domain of mathematicians that became its own offshoot. We developed the first computers, the first languages, and most of the greats of computer science were actually mathematicians. Just being able to program will give you a serious edge, so do not discount this.
Go to school anywhere they will take you my friend and don't look back.