When a mathematical notion is said to exist, it seems that there is a lot of freedom in what this particularly means, a freedom which accounts I think for why some people are finitists or ultra-finitists. How would you define mathematical existence?
The short answer is that mathematical existence is always relative to a system. If you are working in the realm of positive integers, then only such integers exist. But mathematical systems can always be extended to a larger system in which more things exist, and often these systems are developed to prove results about the original, smaller, system which includes just those things that we are primarily concerned about. The things peculiar to the larger system may be variously regarded as nonexistent abominations, convenient fictions, or perfectly good entities as long as their definition is consistent. Most mathematicians take the latter view. Then the question arises: how far can this system-extending process be taken? We can systematize this question by considering that most of mathematics can be formalized within set theory (I would say all mathematics, but that's not universally accepted). The standard formulation of set theory is ZFC. Some people think that's already too much (in particular, they are not happy about the "C" part), most people go along with it, and some are intrigued by the possibilities of extending it. A major direction of this extension research is in positing the existence of what are called "large cardinals". Although each class of large cardinals was conceived originally by specific properties rather than by reference to the others, it turns out amazingly that, as far as is known, the cardinals form a linear hierarchy of existence: if you believe in one, then you can be assured of the existence of the cardinals below it, and you can consistently disbelieve, should you be so inclined, in the existence of those in a higher class. Large cardinals are not all as fanciful as they may seem: the existence of some of them is equivalent to the truth of some relatively down-to-earth-looking mathematical statements (Harvey Friedman is the big name in this area.)
The next step is: how far can the large-cardinal hierarchy be extended? Here I think it all gets blurred. And perhaps it's not reasonable to always expect finite answers to questions about the infinite.