Is it mathematically wrong to say "infinite number"? I often hear the phrase "an infinite number of..." in mathematics. Is this phrase mathematically ungrammatical, since infinity is not a "number"? 
I'm sure some people will say that whether this phrase is right or wrong is a matter of opinion or taste, or that if people are saying it and know what it means, then who's to say it's wrong. Fine. I'm just curious about how most mathematicians feel about it. Could it lead to conceptual pitfalls? Or should the most modern and broad notion of "number" include "infinity" as a member?
 A: "An infinite number of..." can be understood mathematically as saying that the set of these things is infinite. This is all. 
The human languages, in general, are not formal or literal. Indeed "almost all" expressions of human languages are metaphors, and many expressions that seems not metaphorical today were metaphors many centuries ago.
A: 
Is this phrase mathematically ungrammatical, since infinity is not a "number"?

No, but not for the reasons you think. What is infinity?
Infinity can be taken as a cardinality - an "amount" of something, in a very, very loose sense. Mathematically, you can look at a set, look at its elements, and say it has a cardinality - a cardinal number, which measures how many elements are in the set.
For example, the set of $\{1, 2, 3 \}$. Well, we have three elements. Thus, this set has cardinality $3$.
What about the integers, the set $\mathbb{Z}$? Well, there are obviously infinitely many of them: if there were finitely many, there would be some "least" or "most" integer $n$, but I could give the integers $n-1$ or $n+1$ respectively. Thus, there are infinitely many integers.
Let us sidetrack for moment and discuss next the cardinality of the real numbers, the set $\mathbb{R}$. How many real numbers are there? Well, again, the same argument could show there are infinitely many, but we run into a problem. We can show that, in a weird sense, the infinity in this case is bigger than that of the preceding example of integers.
To touch on this we want to formalize this notion of "infinity" and "cardinality." In particular, we want to note: if there exists a function which is "bijective" between two sets $A$ and $B$, then the sets have the same cardinality. What is a bijective function? It is a function which is "injective" and "surjective" - basically, if $f$ is our function, then $f(x) = f(y)$ will immediately imply $x=y$, and for every element of $B$, there is a corresponding element of $A$. That is to say, $f$ has the properties that unequal inputs go to unequal outputs, and every output (in $B$) has a corresponding input (in $A$).
You might also know that bijective functions are invertible, but that is getting bit beyond this scope.
This notion also holds for infinite sets: for example, this is where the notion that there are just as many natural numbers as there are integers or rationals comes from: you can establish bijections from $\mathbb{N}$ to $\mathbb{Z}$ or $\mathbb{Q}$ (the set of rationals): thus, the same infinity governs each in terms of cardinality.
What about the reals, then? As it happens, you cannot find such a bijection from the integers to the reals. This is shown by Cantor's diagonal argument, a famous proof by contradiction. Suppose you somehow have a list of all of the real numbers: then this list is "countable," which is also true for the integers. Label each number with an integer, establishing a bijection. But, from each of those real numbers, you can take a digit, alter it, and then concatenate them all to get a number not on the list. I'm totally oversimplifying the proof of course (you can read further here), but this is essentially the same as our pseudoproof about infinitely many integers earlier: supposing there are "infinitely many" integers (in the sense of our infinity for the integers), we can find more. This is sufficient to establish there exists no surjective function $f$ from the integers to the reals, and thus no bijection.
Okay, so the infinity for the reals is obviously bigger than the infinity for the integers...
At this point we introduce some names $\aleph_0$ is the name we give to the infinity for the integers. We let the cardinality of the reals just be known as the continuum $c$. We have established that there are two "infinities", one which is bigger than the other (in the sense that there is no surjection from sets of cardinality $\aleph_0$ to those of cardinality $c$).

Okay but what the hell is the point of this ramble?
Notice what we started doing. We introduced the notion of "cardinality" as "the amount of something," or more formally as the number of elements in a set. (You could go even more formal but this rant would lose more value by doing so.) We then introduced the notion of an "infinite cardinal" - in fact, we gave two, and demonstrate that one is bigger than the other. These are cardinal numbers in the sense they represent the number of elements of their respective sets, and infinities in that their sets indeed have infinitely many elements.
We could even establish more, higher, greater infinities in this sense.
But more importantly - infinities are numbers. You can do math with them, compare them. Want to add two infinite cardinals, $\alpha + \beta$? Well, that's just defined as the maximum of the two cardinals, and there's rigorous reasoning for that. You can read more about it on Wikipedia.
Infinity is a number, and does represent a quantity in its own right, because these infinities as constructed here are cardinal numbers. At best, you cannot say infinity is a "real" number in the sense of real numbers: real numbers are not necessarily cardinals. It's a different number system, essentially.
So to say "infinitely many" is not at all grammatically incorrect.


...But it is ambiguous.

This rant does have a secondary point, and that's mostly because it comes to a point close to home since to say "infinitely many," on its own, does not imply which infinity we speak of. We could speak of sets of infinite cardinality: do we mean those with cardinality $\aleph_0$ or some greater cardinal? This hits close to home because this exact ambiguity came up in a homework assignment for me once.
In that case, it was imparted to me that "infinitely many" can represent any cardinal number $\kappa$ such that $\kappa \geq \aleph_0$, i.e. any infinity at least as big as that tied with the number of integers there are.
So in a sense, it is mathematically ambiguous and more idiomatic to say "infinitely many", as one commenter noted. In certain mathematical contexts, we will want to be specific as to which of the infinite cardinals we are working with. Sometimes we don't need to be and saying "infinitely many" is fine. But it does touch on the point that, mathematically, language needs to be precise at times to avoid inaccuracies or ambiguities like that.


Perhaps just a more grammatical and less mathematical viewpoint?

I feel it may be fruitful that, after all this, we can also discuss the grammatical implications and intuitions at play with "infinitely many." ... Perhaps it might even be more illustrative. >_>
Hopefully, we can take as given that there is no issue with "finitely many." In that light, consider: what would be the opposite "finite"? That which is finite is infinite - limitless, without bound.
In that sense, we aren't even considering numbers. We're simply saying there is a limitless number of something of which there is "infinitely many." Of course, which infinity is still an ambiguity, but this drives us away from the notion of whether infinity is a number, in that there's just "not a finite" amount.
A: Infinity is not a number, as you said. However, it is a cardinality. For example, the cardinality of the naturals is $\aleph_0$ and the cardinality of the reals is $\aleph_1$ , basically, different sizes of infinity. Thus, although infinity cannot be considered a number, it can be used as a measure of size. Hence, it is mathematically and grammatically correct to use phrases such as 'infinitely many', or 'an infinite number of '.
A: I think "an infinite number of _____" is a fine use but here's a maybe far fetched potential conceptual pitfall. 
There is an infinite number of ways to answer the posed question. In fact (in some sense, because I am limited to using symbols from a finite alphabet), there is $\aleph_0$. That is, there is countably many different answers one could type up. 
And if you sequence the rational numbers $q_1,q_2,\dots $ then this sequence has an infinite number of subsequential limit points. In fact, there are continuum such limit points. 
A potential pitfall would be to think that these are same. Just because there is an infinite number of ways of doing $X$ and infinite number of ways of doing $Y$ doesn't mean there is the same number of ways of doing $X$ as there are ways of doing $Y$.
Anyway: I think this pitfall is somewhat far fetched but it's definitely worth keeping in mind that the rules that obvious to us when dealing with number do not apply to infinities. 
"If there $z$ ways of doing $X$ and there $z$ ways of doing $Y$ then there are the same number of ways of doing $X$ as there are ways of doing $Y$. Is a perfectly valid conclusion when we are working with $z=17$... but not when $z$ is infinite. 
