I am not sure how 'mathematical' the riddle part of this story is. :) Each guest paid \$9 because together they paid \$30, and got back \$27. Of those \$27, the manager got \$25, and the waitress got the other \$2 -- there is no sense/reason in adding \$27 to \$2 -- though you might have a gut feeling that you are headed in the 'right direction' of getting the initial \$30 that way. But take a look -- there really is no extra dollar left, that's all it should be -- \$25+\$2. In other words, each guest paid 9 dollars with the tip included into that.
Another, possibly more insightful way to look at things:
The guests initially paid 30 dollars. The waitress returned them 3 dollars. So the guests ended up paying \$27 (not thirty!). Of those 27 dollars, 2 dollars were pocketed as a tip by the waitress, so you could also say that they paid \$8.333... each and then added a two dollar tip. I think it's a question of order of operations (and using it with respect to appropriate quantities) that could be confusing here.
Punchline: the waitress' 2 dollars were part of 27 dollars paid by the customers. They saved the other 3 dollars of the initial \$30 because of what the manager said, so if you are wondering 'what happened to \$30?' it's more of 27+3, or 25+2+3, where \$25 is what they paid without a tip, \$2 is the tip, and \$3 is the amount they saved.