Besides seminars, what other social outings are there for mathematical hermits? I'm assuming I can attend a seminar without a fee.  I already checked into auditing courses.  I used to think that was free, but apparently you have to pay full tuition costs to do that nowadays.
I checked meetup.com.  There is one math group in my area, one event, and 2 people attending.  I will not attend those nor will I be checking meetup again.
MSE is pretty social, but I would like a real life experience in all 4 spacetime dimensions.  
Thank you!

Also, first I have to find a seminar with topics that I could begin to grasp.  So what's the simplest way to do that?  I'm thinking: list all universities within 50 miles from my home, then go to their math department websites and view the upcoming seminar topics.
^^
Actually I just tried this and it's not as cut and dry as I first thought.  You have to navigate and search around to find 0 seminars on typical math subjects.  Is there a global list of seminars in my area or something?

Okay, I've temporarily given up on seminars.  The only place with a decent list is UC Berkeley, and I'm not near there.  I bet if I showed up, they'd turn me away because I'm not a student.  
Maybe I'm looking for a conference, a workshop?  I'm not even sure any more.  I will just hermit away I guess.  :)
 A: Assuming there's a reasonably sized research university within easy driving distance of your home, I would recommend finding a weekly seminar in an area you are interested (and have some background) in. You won't understand everything that's said in every lecture of course, but over time you'll understand more and more. (Ravi Vakil has some great advice about getting the most out of seminars in the On Seminars section of this website.) At the end of these seminars the attendees and speaker will often go out for dinner, which will add to the social dimension. Although it might be awkward inviting yourself to dinner when you're the only one not affiliated with the university and no one knows you, this too will become much easier over time as you become a regular at the seminar. If you don't have a strong interest or background in any particular area I'd advise you to check out the department's weekly colloquium. These are meant for general mathematical audiences and are often accompanied by a tea, where you'll be able to socialize with professors, graduate students, etc. I should add that all of this should be completely free, regardless of whether or not you are affiliated with the university. Finally, if you're having trouble finding a seminar to attend (seminar names can sometimes be quite difficult to interpret), I would recommend attending weekly colloquia and getting a sense for which professors are inviting which speakers. (Sometimes this will be posted online. Other times you'll have to see who introduces the speaker at the colloquium.) If you're at a great colloquium talk in which the speaker was invited by Professor X, then I would go to Professor X's biographical blurb on the department website to see which research group Professor X belongs to. I would then start checking out that group's weekly seminar.
A: Most university level maths courses nowadays are service courses run for the benefit of other faculties, such as engineering or finance.  Apologies for the apparently irrelevant beginning, but read on...
A consequence of this is that a large proportion of maths students are not really all that interested in maths.  Most are mature enough to realise that maths is important for their future and will study it diligently, but that's as far as it goes.
A consequence of this is that many maths academics are delighted to meet students who are actually interested in the subject, as you appear to be.  So I would suggest that you don't yet give up on auditing courses.  It may be that officially you would have to pay a fee, but it may also be possible that you could find a lecturer who doesn't mind you quietly sitting in on classes and not mentioning it to the institution.  If you do this, please be sure not to take advantage of the lecturer's kindness: most academics nowadays lie somewhere in the range from "very busy" to "grossly overworked", and may not have the time to do anything for you other than let you sit in on their classes.  On the other hand this may not give you the social contact you are seeking.
The problem with seminars is that they are often very specialised.  (You have given no indication of your background so I don't know if this would be a problem for you.)  Even top level mathematicians will rarely attend a seminar outside their own area of expertise, because they know they are unlikely to understand even the beginning of the seminar.  On the other hand, if you find somewhere with seminars that sound accessible to you, just contact someone and ask if you can attend.  What's the worst that can happen? - they say no.  What's the best?
Good luck!
