I am so upset that I want to punch him in the face
This would be an understandable (although inadvisable) reaction. Here are a few responses I recommend you give to him:
"Define what you mean by 'exists'."
"Define what you mean by '1'."
"Define what you mean by 'distance'."
"Prove that 1 exists."
"Prove that the distance from 1 to 2 is 1."
"What is the distance from 1 to 1?"
These questions are designed to get him a) actually thinking about what he's saying, and b) quiet.
In response to "Define what you mean by '1'" and "Prove that 1 exists", he will undoubtedly point to "one rock" or "one bird" or "one airplane", etc., and then you can respond,
"Ah, but those are all one of something. I am talking about the number 1." Here is where he will be a bit confused. The number 1 is a human abstraction from our experiences with one <object>, but it does not exist as a physical object. Here is my favorite quote regarding this (from Linear Algebra by Fraleigh and Beauregard):
"Numbers exist only in our minds. There is no physical entity that is the number 1. If there were, 1 would be in a place of honor in some great museum of science, and past it would file a steady stream of mathematicians gazing at 1 in wonder and awe."
There are several fundamental issues that your adversary is missing:
Firstly, there is no agreed-upon definition of what it means for something to be a "number". If someone said "I am only going to call even integers "numbers", the odd integers don't count", they are just as correct as someone who calls every mathematical construction humans have ever come up with a "number". Mathematicians have decided on definitions of "integer", "complex number", etc., and the statements that "0 is an integer" or "$\sqrt{2}$ is not an integer" are true because "integer" actually means something. The word "number" is a vague word that has no mathematical content.
Secondly, mathematics is a human construction:
"The mathematician is entirely free, within the limits of his imagination, to construct what worlds he pleases. What he is to imagine is a matter for his own caprice; he is not thereby discovering the fundamental principles of the universe nor becoming acquainted with the ideas of God." - John William Navin Sullivan
"Mathematics is a game played according to certain rules with meaningless marks on paper." - David Hilbert
"[Mathematics] is an independent world/ Created out of pure intelligence." - William Wordsworth
So, what "exists" and what "doesn't exist" depends entirely on what we've set up for ourselves. Mathematicians pick a certain set of axioms, then follow them to logical conclusions. It makes no sense to say, separate from a previous decision about what axioms we're using, that "0 doesn't exist". Only after you've decided what axioms you are using, and what exactly it is that would characterize the object "0", can you decide whether your system includes such an object.
If you really want to get at him, here is a recommendation: Tell him, "I am going to prove directly to you that 0 exists." For several seconds, pretend to be deep in thought, trying to remember the details of the proof. Surely he will look on with anticipation, eager to attack whatever you are going to present. Then, appear to have recalled the key step, and very slowly and deliberately draw a massive "0" taking up the entire piece of paper. Look extremely pleased with yourself, as if you had made a difficult and convoluted argument.
The fact is, that is exactly what "0", or indeed anything in mathematics, is: a symbol we write down. What "0" means in mathematics can vary widely; perhaps it is the zero element of $\mathbb{Z}$ (the integers), or the function from $\mathbb{R}$ (the real numbers) to $\mathbb{R}$ that is the constant 0 function. Within a given context, e.g. specific axioms and structures, it makes sense to decide what properties an object you want to call the symbol "0" might have (for example, if there is a notion of "addition", you would probably ask for an object "0" to satisfy "$a+0=a$" for all objects $a$), and then, having decided what "0" should refer to if it did exist, actually find out whether it does, within your axioms / structures. But without deciding what "0" means beforehand (and remember, 0 isn't something like "no airplanes", since physical objects are irrelevant), all "0" means is the symbol 0, and it is quite easy to prove that that exists - just draw it!
Of course, this won't be satisfying to your adversary at all, but I guess I'm just trying to think of ways you can get your just revenge by making equally absurd arguments :)
Good luck confronting him, perhaps comment back to let us know how it goes!