This is actually a very good question. For those of you who think there's nothing about being deaf that prevents you from doing math, this may very well be true for those who are heard of hearing (meaning, able to communicate through speech, but with some difficulty). It may also be true for those who become completely deaf at a later age, though I don't know much about that.
However, for those who are born deaf and grow up with American Sign Language (ASL) as their only means of communication, there are definitely some obstacles. First, ASL does not simply mean taking an English sentence and replacing each word with its sign-language equivalent. It has an entirely different grammatical structure from any spoken language, and for that reason, many Deaf people (the capital D is used to denote those whose primary language is ASL and who are part of Deaf culuture), who have never heard English spoken, do not achieve complete fluency in written English, though many achieve proficiency.
This means that Deaf people who want to do math cannot study it in their own language. And if they want to study math in English, they have to learn English without ever hearing it spoken, and more difficult still, without ever having another spoken language to compare it to. Again, many Deaf people have learned written English well enough to communicate, but it's not a first langauge. Add to that the fact that a Deaf mathematician would have a very difficult time speaking to a mathematician who did not know ASL, and would probably only be able to do it in writing.
You'll notice none of the people mentioned in the answers were born deaf. And being born deaf is decently common. It's a recessive gene, but it's hereditary.
I'm not saying it's impossible, and I don't really know how difficult it is or isn't, since I'm not Deaf, but I can certainly at least see some reasons why it could be difficult. So if I heard of someone born Deaf, who never heard a language spoken, and who nevertheless became a mathematician, I'd probably be every bit as impressed as I would be at hearing of a blind mathematician.