To answer your question directly: no, infinity is not a number. That is, for most people in most situations most of the time, infinity is not included in the set of numbers.
Qiaochu gave a very good answer and I agree with his opinion parts. As Qiaochu pointed out, it depends largely on which definitions you choose to accept, or use, at any given time. Sometimes it's useful for infinity to be a number, other times it isn't. When Qiaochu talks about rings, sets, fields and spaces, and when I say "sometimes yes, sometimes no", what we're doing is referencing a group of axiom sets, or rule sets, which has different definitions and rules as to what numbers and infinity are.
If you're interested, this topic bleeds into a discussion about the history of math, mostly the last 200 years. Basically, the idea that any given definition or rule is "absolutely right" or "absolutely wrong" was discarded in favor of a more axiomatic approach. That is, things may be one way one day, and another way another day; and we, as mathematicians, are ok with that.
For a quick concrete example, ask yourself, "Is 2.5 a number?" If you're counting people, it is often absurd to consider 2.5 a number (thus the joke in the T.V. show). On the other hand, if you're measuring mass, it may be absurd to ignore 2.5 as a number. So the question, "Is 2.5 a number?" has no one answer all the time. In the same manner, the question "Is infinity a number?" has no one answer all the time.
Some interesting highlights on the history of math and infinity:
-Alice's Adventures in Wonderland, by Lewis Carroll, is supposed to be a sarcastic take on the emergence of modern math, notably Non-Euclidean Geometry and Abstract Algebra. Carroll insists that math has was fine for the last 2000 years, and it doesn't need to be adjusted. He portrays emerging math concepts as absurd nonsense in wonderland.
-Georg Cantor rocked the boat with his work on infinity.
As far as describing it in an intelligent way, just say that infinity is not a number because infinity is a meta word not in the set but used to describe the set. Just as the words "unbounded" and "non-empty" are (usually) not considered as numbers, infinity is (often) not considered as a number.