Mathematicians who started late, similar to my situation? I came here since I know this is the best place to ask a question.
I'm a first year student who changed his major to applied mathematics.  In middle school I was a garbage math student, but I realized the importance of math in high school when I was introduced to amazing teachers who truly loved what they did.  I put myself in tougher classes and eventually got to AP calculus.  There was an error in my idea, I never really got a deep understanding of the stuff I was doing and was struggling since I didn't understand the basics and never really did practice problems.
This year I began to start over from scratch from pre-algebra working to pre-calculus. Even though I have already took Calculus.
I'm in a introduction to research class this semester and we are preforming a meta-analysis of some random topic and then presenting at the end of the semester.  I'm really enjoying it, and I will definitely apply for more research as I progress through my undergraduate career.  (Urge to Compute)
I know I'll probably never win a field's medal, but I'm really intimidated and humbled by the near perfect SAT math scores and Math Olympiad participants.  
It's too late for me to have that, but the best quality I have is sticking with the concepts and problems until I can explain them to my dog. (Basically until I understand it)
I'm really sorry for the long post / soft question, I've just been thinking about this since 11th grade but never asked anyone about it. 
Basically I'm just wondering if I'm wasting my time, and if there have been mathematicians that were in a similar situation.  (Famous or not.) 
Again, sorry for the soft question and thank you for taking the time to read this!
 A: I think I'm in the exact same situation as you. Expect, when I was ~6 I'd love playing with numbers and stuff, just trying to figure out ways to add large numbers and eventually discovering some stuff I later learnt by myself. Come high school, I was total garbage at maths. Absolutely trash tier, until I started taking it seriously and realizing how important it is. From then on, and once I got a good teacher, I started to really like math but I was extremely far behind that it was so unmotivating spending hours everyday to try and do what others got in a couple of minutes, since I was trying to catch up. I am basically in 11th grade now, and I am certainly not as good at maths as I'd hope but I absolutely love doing it. I'm just not good at it, at all. I don't know why.
A: Let me tell you that it's not too late to enjoy the field of mathematics.  I too went through a circuitous path of getting into mathematics.  I was extremely lazy in high school, had mediocre SAT scores and had no idea what a math competition was.  I was a business major for undergrad and studied finance.  It wasn't until after I graduated that I made friends with a guy that had been a math major in his undergrad.  He eventually convinced me to take the honors calculus series at a local community college and promised he'd help tutor me.  I was fortunate, my teacher there was a PhD student at a local well known college and took a liking to me and became somewhat of a mentor.  I was a little neurotic and became obsessed with math.  I ended up quitting a full time job, moving back in with my parents and then taking classes at the local university (you could pay and take classes as a non-matriculated student).  I did this for two years, taking only math classes and then eventually applied to their MS program and was accepted.  I have since scored in the top 97% percentile of the GRE general math, and published one paper with another in progress.  I'm going to take the Math subject exam in the Fall and apply to Phd programs.  
So, will I ever be a math wizard... no.  But do I love mathematics, yes.  I share this with you so that you know, there are people out here that started even later than you and are happy they did.  
I'm not sure where you are in your education, but working problems is one of the (if not the) best thing you can do to solidify your foundation.  The book that made me want to become a mathematician: 
http://www.amazon.com/The-USSR-Olympiad-Problem-Book/dp/0486277097
A: Don't know if this helps but if your aim is to enjoy mathematics, and if you are enjoying it, then that sounds like success to me!!  Hang in there, you are not wasting your time.
A: I can relate to your feeling of being intimidated by other people that seem so far ahead of you, and the sense that you might never be able to catch up.  Piling on top of the natural, unprompted feelings of inadequacy, from time to time you will run into professors and/or other students who are into what I've heard called "mathismo"--deliberate effort to prove "I'm smarter than thou".  Such things are a fact of life.  It helps to put a name on it--"that's just mathismo, I'm going to ignore it"--because you can realize that focusing on this will get you nowhere and in fact will impede your progress.
You don't need to worry about whether you'll be able to prove important theorems or whatever.  Not yet.  You have only seen a tiny little bit of the field right now.  There is so much you can do with math.  One thing that sticks out to me, based on your description of yourself, is actuarial work. Great pay and high job satisfaction, and it really fits with your description of yourself as doggedly going through the basics until you understand it really well, etc.
But that's just one possibility.  Math is a gigantically varied subject--"set of subjects" would be more appropriate.  You are very likely to be able to find a niche that you enjoy.  If you like what you are doing so far, I would say that you should keep doing it.  Maybe something you end up doing will lead you in another direction eventually.  That's not failure, that's just life.  If you are really making sure you understand it as you go, you'll be able to use what you learn in math for the rest of your life even if you switch to another major.
My advice is "Onward!".  Keep at it, and if it gets to the point where you just aren't enjoying it any more, do something else at that point.  There's no point in worrying about the unknowable--you can't know right now what you're going to think about upper level math, because right now you probably haven't really seen anything like it.  School unfortunately just doesn't give people a very clear idea of the vast scope and depth of what mathematics has to offer.
If this is what you think you want to do, don't let anything--whether it's an inadequacy in your past or a current deficient comparison to your peers--stop you.  At the same time, be honest with yourself in realizing that you can't, at this point, really know where it's going to lead.  Mathematics is an awesome universe with tremendous opportunities for exploration.  It's a great place to be looking around and taking things in.  Enjoy the journey.
A: In middle school we used to complete math problems given by a computer system and then based upon our answers we would be promoted to a new level of problems or stay at our same area. I never made it to algebra until high school. However once I began studying algebra, I would do my homework in the minutes before class started and generally do well. I still felt incompetent in math. I did okay in some course and well in others. Algebra was my favorite as I would play on my calculator, graphing functions in order to see what they looked like such as high order polynomials, etc;. I never liked, and still don't particularly enjoy trigonometry. However I made it to calculus by my senior year. The summer before my senior year, I was at a smart kid camp and instead of hanging out and making friends like most of the others, I spent my time in the library and in my room learning about the concept of a derivative and an integral. Calculus changed my life in terms of my understanding of the power of mathematics to describe many aspects of our reality in such a beautiful and succinct manner. I attended college and immediately began as a finance major. However we were only required to take two high level math courses in calculus and applied mathematics such as system of equations (no matrix theory). I rejected doing the bare minimum and instead I opted for engineering calculus, which I found to be much easier than most people who attempted to talk me out of it did. I stopped listening to others after that and went after what interested me, mathematics were intimidating at that point but my curiosity to learn more about the subject would not go away. I proceeded to take calc 2 (integrals/series/sequences), calc 3 (multidimensional calc), and calc 4 (ode's) and I have fallen in love with the field of analysis. Since then I kept taking various math courses in probability theory, optimization, discrete mathematics (as I'm technically a computer scientist), etc;. I feel that I didn't truly begin to learn mathematics until the age of 17, I have never gone to math camp, competed in math competitions, but I have a love for math that is untouched by any other subject as it is present in nearly everything I have ever studied. Who cares when you start, if you have the passion for it nobody can take that from you and nobody can deny the beautiful truths that are consequences of  mathematical rigor and analysis.
