How do you respond to "I was always bad at math"? Here in the U.S., it is my experience that over 75% of adults I meet socially will volunteer that phrase or a variation upon learning that I am a mathematician.  I find this frustrating, since almost nobody would  brag about being bad at history or English or really any other subject.   Normally I change the subject, or perhaps say they must have had a bad classroom experience, but that feels unsatisfying and turns the conversation in an ugly direction.
Surely some of the bright minds here on Math.SE have found a good response. Note that I'm not looking to  alienate a new acquaintance, so let's keep it positive.
 A: I was never good at singing till a stranger at a wedding turned to my wife after the service and asked her "who was that bloke with the beautiful voice standing next to you?"
I once helped a person with their last chance at GCSE (high school) - they needed to pass to become a teacher. Looking at their work, they could do what they needed - indeed, over the three exams they'd failed, they'd shown that they could answer every question they'd been asked. They got a 'B' when they needed a 'C' (this was a fortnight before the exam - so really last chance saloon). I said "bottle that feeling and give it your your pupils".
People have been educated to believe they are not capable - most are. The most powerful thing is to catch them doing it, because that has the potential to break the spell.
A: Math is hard for everyone. It takes long hours of practice to become competent. I think that's what people want to hear; and fortunately, it's true.
This reply has the advantage of steering the conversation in a somewhat more positive direction---in my experience, usually towards whatever it is that the person does like to do and has practiced long hours at. This is usually more interesting than having a conversation about poorly taught math courses.
A: Asserting you are a mathematician without immediately volunteering further details is just a bad thing. It's very closed.
The principle is simple: As soon as you mark yourself as part of a world that people are not part of and don't understand, you alienate them. Unless you give them something that relates to their own experience, you are going to have a hard time.
Consider this conversation:

— Yeah, I'm a school teacher.
  — Cool, do you teach little ones or big ones?
  — Little ones, they're constantly surprising...
  — Yeah, I've got two kids about to go to school.
  [Conversation ensues]

It's OK for a teacher, because everyone knows what a teacher does, roughly. Even if they don't really understand what a teacher does day to day, they know why they do what they do.

— Yeah, I'm a mathematician.
  — ... (blank look as they realize they know nothing about what a professional mathematician does) ... Yeah, I was bad at maths at school. (nothing else to say)
  [Awkward silence ensues]

In this case they have nothing to work with. With something like maths, you have to give them something extra to relate to, or they will just jump to their only experience of mathematics: school, where more than likely they were not working at the standard of a profession mathematician.
It is easy to find yourself in the Ivory Tower. People responding in this way should be taken as an indication that your work has become very detached from social reality — although some mathematicians think this is a virtue, it really is not, it's more like laziness.
Things you could say instead of "I'm a mathematician":


*

*"I investigate the properties of prime numbers."

*"I look at how fluids move."

*"I'm interested in symmetry."

*"Do you know X? Well, I do something like X."

*"I try to find out why Y."


Just give a more open answer that give people a chance to grasp what you do. Do not require them to be part of your world to understand it. If you do this you will find people will suddenly become quite good at maths.
A: I respond with "what was your favorite subject?"   If I was bad at it then I would say so.  If I liked it even though I was bad at it, I would move the conversation to that topic.  If I didn't like it, I would say so and ask "why was it your favorite subject?"
The "what do you do?" question is an attempt at finding a commonality and a mutually interesting topic. Keep moving in that direction.  If the disclosure came in some other context, well, I'd just say "most people don't like math" and go back to the previous topic.
A: I get a variant of this depressingly often when people find out I am an engineer. You might be surprised how many people think "engineering = math."
It's worse than you suggest. Being "bad at math" has actual consequences in life in ways being bad at music or history or poetry do not. Being bad at math means you likely make foolish decisions, especially when it comes to finances. This is a problem because people who say "bad at math" normally mean "I avoid math at all costs but wish I was better at it."
So what can you respond? Realize people mean something different. They are really saying, "I dislike math and/or have no interest in learning but wish I magically knew it" - not "I am bad/was always bad/could never do math." 
You can approach this a couple ways.


*

*"Why do you say that?" is generally how I address it as this leads to lots of followup questions

*"So was I" can make for interesting responses

*"It takes a lot of work/practice"

*"Oh wow! how do you maintain your life?!"  (I don't recommend this)


Many of these people have been conditioning themselves for years or even decades to convince themselves they are "bad at math." 

Just a note, this used to bother me or even make me feel guilty, too. There is an implicit, "aww, man, I wish I was as lucky as you were to be good at math" in that response.
But at the end of the day, I've had to realize a lot of people have simply spent years telling themselves they are bad at math. That's not my fault. 
Don't feel guilty by people saying a variation on "I was bad at math."
A: I tend to nod agreeably and say that math isn't instinctive and takes a lot of time to learn, but that I was fortunate to have positive early experiences with math.  Maybe I'll say a few words about what I think were key events (or key things that didn't happen--like never had a teacher who turned it into confusing drudgery), suggest that most people don't have as good of experiences, and suggest that teachers and early reinforcement is really important.
That leaves the conversation in a nice flexible position: I haven't tried to make the person feel stupid, I haven't claimed that math is hard for me (it was nonintuitive to learn, but it was fun and pretty easy, so it was no problem playing with it until I developed some intuition), and I've given easy paths to talk about either the positive aspects of math, the problems with math teaching, or any positive early experience the other person might have had.
The main problem is saying too much: the person probably wasn't looking for a thesis on math ability.  I haven't figured out how to do this briefly enough to flow quickly in casual conversation.  (But I haven't had much practice either--most people I talk to are not bad at math.)
A: Since a vast majority of people are under the impression that mathematics is like the curricula they had in primary and secondary school, this is not too surprising.
Really they don't have any qualification to say they are bad at math. (I know I didn't see any real math until college.) They might have been bad at the curriculum, but that's understandable given how boring and stupid it is. That is what happens when you teach mathematical tasks as behaviors to be trained in. 
Unfortunately, teaching mathematics in an effective way would be more time consuming and less testable than the current curriculum. Moreover, very few people in the position of teaching those age groups are probably not suitable for teaching real mathematics. In short, the way people expect young people to be educated is at odds with actual effective instruction. 
For anyone who hasn't already read it, Lockhart's lament is relevant to this and is worth reading once.
A: At the point where you say that you're a mathematician, that is then the most recent topic thrown into the "conversation pot", and the only connection most people have to that topic is that they were bad at it once upon a time.  So that is all they can say, if they say anything at all.
That person you are speaking with was bad at math at a time when they had to work with math on a daily basis, and of whatever they did learn, they probably remember a quantity that, in the vocabulary of those who were bad at math, is expressed as "jack squat".
So, that person having revealed everything they know about math, that topic is thereby exhausted.  At that point, someone's conversational skills will have to kick in and move things in a different direction. One principle that works very well in conversation is "Me too, I'm just like you". Well, not said literally, of course.
"I'm a mathematician"
"Oh, man, I was bad at math in school"
"Believe it or not, so was I! I ended up in this remedial class, even. That's where things turned around for me because there was a great teacher who really helped us to "get it". Still, I had no idea at that time I'd end up an academic, let alone in math. Life is funny that way."
Just be glad the response isn't this: "Oh, you're a mathematician? Like Euler and Gauss and those guys? What have you discovered in math that people are using to solve problems?" 
Now you're screwed in the conversation.
A: Sometimes, the conversation never gets as far as "I was always bad at math." I was at a party with my wife while we were in graduate school. A guest  asked her what she did, and my wife replied that she was a math teacher. The guest then turned and walked away, never saying a word in response, and never to be seen again. I think some people have some deep- seated issues with math, not just with mathematicians. Why, I do not know, and it would be useless for me to guess, as I am not really qualified to do so.  
A: To put a slightly different spin on this (before @Zev votes to close :) ). When folks ask me what I teach, I ask them to name their least favorite subject. Occasionally I do get English or history or science. But 90% of the time I get math ... to which I reply "bingo." :) I then engage them in a discussion about how math (generically) and history (for me and many others) are ruined for the masses by secondary teachers who just don't get it and who teach relying almost totally on rote memorization/manipulation. They concur 99% of the time on this. At some point, I reassure them that there are some really good teachers out there (and I claim to be one) :P
A: Just ask what the student likes to do or does well. Then show that there is math in it. Then you can use this as a base to elaborate and cultivate an interest.
A: I think it depends on people. Beyond practicing, it's about logic, and sometimes about abstraction. I think a lot of people would find hard to put effort into it, because abstracion is a totally irregular way of thinking for an everyday people.
Just like anything else, math can be learnt anytime by anyone. Just have to begin it!
However, talking about it without any subjective opinions is hard for me, because even though I hate practicing all the time, basically I love logic things, such as math (but I prefer programming) - overall, feel free to downvote me if I'm wrong.
A: I have answered this before by saying "I was always bad at ". But then I would tell them how I got better at something, and ask them why they think they are not good at something. For people who genuinely want to become better or that do have an appreciation for the subject matter, I use it as an opportunity for them to teach me something I don't know in exchange for me to teach them something I know. I like the concept that learning and teaching should come hand-in-hand.
A: Serious math involves serious abstraction.  Most people aren't very good at that.  It takes practice.  But when people say that they're no good at math, I'm not sure that's the problem, because people who say that probably have never gotten that far.
When I look at a page of symbols in an area of math about which I know nothing, it looks like nonsense, it looks impressive, and it looks impenetrable.  But I know from experience that that's how math always looks when you don't know anything about it.  So I'm not afraid of the mysterious pages of symbols.  I know that I can learn it--at least if I find the right textbook, and go slowly, and it doesn't turn out that I will need 7 years of study to get to where I can understand this area of math.  (I also think that the mysterious symbols look cool, even when I don't understand them, but that's just my perversion.)
I think that many people just see the impenetrable symbols and shudder.  And they do this even when they know what each symbol means.  Their eyes glaze over.  I think the key is to stop, take a deep breath, and worth through the symbols one by one.  One probably needs to learn that that's a possibility.
About pride:  I once worked in a menial office job with someone who was obviously good at careful, complex reasoning.  Any time I would point that out, he'd insist that I was wrong, and that he was no good at logic.  No, he was an artistic type.  Math is probably like that.  Math seems rational, rigid, inhuman to some people.  So if you're good at math, there's something a little bit wrong with you, from that point of view.
A: I usually say something about math being just another language, and that it takes a lot of exposure, practice, and work to become fluent. A very different sort of language, but analogous nonetheless. This usually gets people interested, because it's not a way that they are used to thinking about math.  I don't know how widely this resonates with other math types, but it feels this way to me: whether or not it ought to be called a "language" I don't know, but in terms of how our brains learn it, I think the analogy holds.
Part of why this helps I think is to move sideways from the usual directions, like math is adding up big lists of numbers, or math is beyond the grasp of normal minds, and relating it to something most people have experience with and a whole different set of conceptions about.
A: I am not a mathematician just a lay person who find abstractions easy to work with.
I reply and tell them that it's not their fault. The problem is that maths is about abstract thinking and all it takes it so get the right abstraction (or model) in the mind of the person and they'll get it.
I tell them that some people 'can visualise maths' easily, but others may need help in getting to the visualisation. It's down to the teacher to help the student 'see the maths' and that if they choose to ignore the fact that they suck as maths and imagine that they've just not had a map that lets them navigate that world easily, they could do so. It's just mid set.
A: Finding some practical application for math can get people quite a lot more interested; helping them to feel like they have a mastery of the numbers; and thus of the world around them.
That most likely sounded like a paragraph of BS though, so what I'm really trying to advertise is the LOGO Programming Language; the story of Seymour Papert's Lisp-like language for small children is filled with numerous instances of kids who were "never good at math" finding that they had really just been keeping away from it due to adverse learning experiences. Here's the book that I did my final term paper on. Here you have kids below 3rd grade, learning systems of geometry, degrees, and even programming idioms like subroutines, simply through the ability to experiment with them.
Applying those kind of ideas universally in a school system would be a huge undertaking, but it could apply very well in a private learning environment. Also: LOGO is old as disco, but the basic principles mentioned in the book are available in the 'turtle' library of Python.
