What would be the advantage of accepting non-measurable sets?
I personally feel that non-measurable sets only exist because of infamous Banach-Tarski paradox...
One correction to your question is that non-measurable sets actually proved to exist by Vitali in 1905, his construction of a non-measurable set is now called a Vitali set (assuming the axiom of choice). The Banach-Tarski paradox appeared about two decades later in 1923.
There is no immediate advantage in accepting the existence of non-measurable sets. In fact it "harms" us in some way, it means that we have to be more careful in how we define measure and so on.
However there is a great advantage in accepting the axiom of choice, or at least the ultrafilter lemma (which is a weakened version of choice), both implying the existence of non-measurable sets. In fact much weaker claims than the axiom of choice imply the existence of non-measurable sets. To name a few:
To read more, you can try Herrlich's wonderful chapter about measurability in his book The Axiom of Choice.
Whether or not to accept such existence boils down, in essence, to what you are trying to do. If you want to do finitistic mathematics, deal with finitely generated objects and a limited collection of their subsets then there is no harm in not assuming the axiom of choice.
However if you wish to deal with infinitely generated objects, such as $\ell_2(\mathbb N)$ or other measure theoretic necessities, then the axiom of choice is usually needed to allow a "smooth" transition from finitely generated objects to infinitely generated objects.
The key problem is provability, a lot of properties depend on the axiom of choice and we simply cannot prove their truth value without it. So you end up having to assume a lot more than simply saying "assume choice". In this aspect, assuming the axiom of choice helps both to decide a lot of properties (but not all, of course) as well allows immediate generalizations of the proofs to higher cardinalities.
To read more:
I'm sure you will get very competent answers discussing the relation between non-measurable sets and the axiom of choice and the like and the reltion between finitely additive and $\sigma$-additive measures.
Here I want to give a fairly practical reason why it might be preferable to have a measure not defined on the whole power set but a smaller $\sigma$-algebra.
There is no intuitive notion of measure or volume for arbitrary sets of points. We do have, however, a fairly good notion of volume for certain geometrical objects, such as rectangular blocks. It is natural to extend our notion of volume from elementary objects to more complicated objects by approximating the more complicated objects by simpler objects. For example, we can approximate the volume of a ball by approximating it by the disjoint union of many very small cubes. Now there is no reason why we should be able to approximate every set of points meaningfully by simple objects we know the volume of. So if we want to assign a notion of volume or measure to every set, we are going to have to make soe ad-hoc choices.
The argument has even more bite in probability theory. Say we want to describe a probability measure on the real line. The usual way to do this in practice is by specifying a cumulative distibution function, which essentially pins down the measure of each interval and then using a result that says that a probability measure on the Borel $\sigma$-algebra is uniquely determined by its values on each interval. If we want to assign a measure to every set, we have to specify lot of values for sets that do not even occur in practice.
When you make a set-theory that pretends that non-measurable sets exist, you are required to treat random variables differently from ordinary real numbers, so they become second-class citizens of the real number line. What I mean is that you need to keep careful track of all the theorems you know about real numbers, because some of these theorem are "random-ready", and can be applied to random variables, while other theorems are "not-random ready", and cannot be applied to random variables.
When dealing with random variables, you need to restrict yourself to only use the "random-ready" theorems. This is an impossible headache for probability, because everyone who works with random variables imagines that they can take on actual values, that you can actually choose the random numbers they describe in the limit, by throwing coins or doing Monte-Carlo, and then, after the values are chosen, you automatically think you can apply any theorems that you know about real numbers to those random values.
When non-measurables are lurking about, you can't do this, and you need to separate your theorems into two kinds, the ones that apply to random variables and the ones that don't. It becomes inconsistent to view random variables as actually having real-number values, because any precisely specified real number has more operations defined on it than do randomly generated reals. So you need to think of random variables as always imprecise, as only having been specified at finitely many digit positions, and having infinitely many digits still unspecified. This is an overly constricting requirement on the imagination, you will screw up without intending to.
To demonstrate the issue with an explicit contradiction, find a basis for R as a vector space over Q (it is a theorem in ZFC that such a basis exists). Let's say you know this theorem, but you aren't aware that it is not random-ready, so it isn't true for random numbers. Now you use standard probabilistic intuition.
You consider two independent Gaussian random variables of unit variance x and y. You consider the sum of these two variables with rational coefficients z=(3x+4y)/5. Now you decide to decompose x into a basis, and call the probability that it is made up of n basis vectors P(n). You conclude that the probability that y is made up of m basis vectors is also P(m), since y is identically distributed, and since the space spanned in Q by n or m basis vectors is countable, you conclude that z is made up of r=m+n basis vectors.
But z is again a Gaussian random variable with unit variance! So z must have distribution P(r) of being made up of r basis vectors. This is impossible (it is impossible for the sum of two independent identically distributed positive integers to have the same distribution). Contradiction. So it makes no sense to decompose a Gaussian random real number into a basis over the rationals.
Where you go next depends on your view of what is an acceptable in mathematical practice. If you think it is ok to forbid a person from imagining a Gaussian random real number as actually existing in the mathematical universe and having values, then you can say "ok, random variables can't have precise values" and at the same time say "every real still can be decomposed into a basis over Q". If you instead believe, as I do, that random numbers should be considered first-class citizens of the real number line, you simply say "it is impossible to decompose arbitrary reals into a basis over Q", and you reinterpret the theorem about decomposing real numbers as applying to a subcollection of reals specified in a more precise way, where the axiom of choice is true, for example, the computable reals, or V=L.
The second point of view is a more correct view because it admits precisely those operations on real numbers which you can idealize as doable in practice. In real life, you can't decompose any given real into a basis over Q, not without a specified listing of all real numbers that you consider as "speakable", like a listing of all computable numbers, or all definable numbers in ZFC, precisely because you can't decompose a randomly chosen real into a basis over Q. The formalization of this statement is in Solovay's model, by which you prove that there are no predicates not involving an impossible-to-specify uncountable-choice function which can define what the basis decomposition of an arbitrary real number would look like. If there were a specifiable basis decomposition, it would have work for random numbers.
In probability, you consider infinite random things as having actual values all the time. For example, when you consider an Ising model on the infinite 2d lattice, you are required to imagine a "randomized configuration" in your head, and you need to be able to consider that such a beast exists as an actual configuration, meaning that there is an infinite list of 0,1 values which correspond to an actual randomized configuration in thermal equilibrium at any temperature T. That's the same as imagining a random real number. Any theorem you prove about all configurations of the Ising model needs to be applicable to a randomly chosen configuration of the Ising model.
But if you have non-measurable sets, you can't imagine that an Ising model in thermal equilibrium has a precise configuration, because there are non-measurable questions you can ask which don't make sense about the randomized configuration. For example, define two Ising model configurations to be "related by translation" if you can translate configuration A to turn it into configuration B. Then define a set of "primitive configurations" to be a unique choice from each symmetry equivalence class, so that any configuration is a translation of some primitive configuration. What is the probability that a given equilibrated Ising model configuration is primitive?
The answer is not zero. The answer is that the question doesn't make sense, the probability is not well defined. You can see this immediately because when you ask "how far do I have to translate the randomized Ising model configuration so it becomes a primitive configuration", if the question made sense, this would define a uniform distribution on the 2-d lattice of spatial translations, which you can't do any better than you can randomly uniformly pick an integer.
You can either, as I do, consider this a rigorous proof that it is impossible to actually pick one element from each equivalence class of translationally related Ising models, or else, as mathematicians have done for a hundred years already, you can consider this as a proof that randomized configurations of Ising models don't exist in your mathematical universe.
So if you are a mathematician studying Ising model configurations, for you there exist constructions which cannot be applied to random configurations, but which can be applied to any specific configuration. Allowing these constructions means that you are forced to view the randomized configurations as somehow incompletely realized at all times.
This is the main difficulty in mathematically defining path-integration on spatially infinite lattices. Physicists not only consider path-integration on lattices, they also consider the continuum limit of such path-integrals, and this requires a further renormalization procedure to take the limit of small lattice-size. Wilson and Kadanoff reformulated the continuum limit problem as a scaling-law requirement on the long-distance lattice correlations, so the real issue is no longer the continuum limit, it is simply dealing with the properties of infinite lattices. Once you have these, the Wilsonian limiting arguments for the continuum are somewhat difficult technically, but relatively straightforward conceptually, and technical difficulties in mathematics tend to get ironed out with time as the language evolves, so long as the conception is clear.
But to consider a Wilsonian picture, you should be able to talk about randomized configurations on an infinite lattice. Requiring that these things exist is the main reason for the divergence of physicist's mathematics from the mathematician's mathematics. When you consider path-integrals on infinite volume lattices (not in infinite continuum, no renormalization), you are defining a measure on infinite random configurations of fields on the lattice, exactly like in the Ising model, the Ising model is in fact the simplest example. You can define stochastic algorithms to equilibrate finite lattices which are uniform for space, meaning you can formulate them as dynamical stochastic processes on infinite lattices just as well as for finite lattices, and these processes equilibrate infinite lattices with the same time-constant as they equilibrate on large finite lattices. An example is Glauber dynamics on the Ising model on the infinite plane. You can define Glauber-like dynamics as an SPDE, and the SPDE with a regularization on an infinite spatial volume will equilibrate in finite time to a random measure on the infinite-volume fields. This SPDE approach with limiting stationary distribution is what is used by Martin Hairer to define the renormalization procedure for quantum field systems.
But in standard set theoretic mathematics, the infinite lattice Ising model, or an infinite volume SPDE, is distinguished from the finite one, in that it is inconsistent to view the randomized configurations of the Ising model as actually realized in the set-theoretic universe, because of the infernal measure paradoxes. For an example of an explicit paradox, consider Hairer's SPDE for the stochastically quantized 3d Landau Ginzburg model on R^3 plus time. The solution at finite regularization is differentiable. Consider picking one differentiable function from each translation equivalence class, and call it a "primitive configuration", and ask "What is the probability density for the translation x,y,z that moves the SPDE solution to a primitive configuration?" Again, this is impossible to ask, let alone answer, as the question would define a uniform probability density on R^3 by translation invariance. This is not saying that Hairer's construction is faulty, it is saying that the construction "pick a function from each translation equivalence class" is faulty, and you can't waste Hairer's time (or anyone else's) by forcing them to consider such ridiculous questions.
You can view Glauber dynamics as defined on infinite configurations, and the properties are continuous with Glauber dynamics on finite volume configurations. But when you extend it to infinite-volume configurations, you are always laboring under the constraint those configurations are imprecisely specified, and the Glauber dynamics is defined only on certain sets of configurations which are specialized. This, in practical applications, is an impossible headache. You can't do probability without imagining, as Hairer often does, that your random variables can take on actual values, that the solutions of the SPDE for specific realization of the random noise are actually differentiable functions. More specifically, you need to use theorems from the PDE literature which are proved for real values and all diffentiable solutions, but now replacing those real values with random variables. If you do this in standard mathematics, there are theorems which don't apply to random variables, you start making mistakes and you get paradoxes. This will never happen in a million years in Hairer's work, because nobody does ridiculous things, like choosing translationally primitive configurations, in intermediate steps of proofs in the PDE literature.
But there is a stronger philosophical position I take. I further say that any "mistake" you seem to make by imagining infinite random configurations and contradicting a set-theoretic theorem is best interpreted, not as a mistake, but as a natural recipe for producing a forcing argument which produces a set theory model in which your mistake is corrected, and the theorem you contradicted becomes false. For example, if I force random Ising model configurations into a set theory using the Ising model measure as a forcing notion, starting with a model M in which there is a list of translationally primitive configurations, I produce a set theory model in which the ostensible list of primitive configurations is no longer complete. This model will have a new list of primitive configuration (including a new one for the configuration I forced in, and all the configurations that can be produced deterministically from this), but you can do this again and again. If you force in inaccessibly many such new random Ising configurations, then the well-ordering of R only "appears" after inaccessibly many steps. You can imagine truncating the ordinals to those below the inaccessible so that the listing procedure won't ever list all the new Ising configurations you forced in, but then the result will not be a model of ZF, unfortunately. Solovay produced a model in which there is no list of primitive configurations of the Ising model, and where the Ising model measure is complete--- i.e. where you can speak about arbitrary random configurations without fear of contradiction, and in a way compatible with the axiom of powerset. In this case, as explained to me by Asaf Karagila, you first force in a map between an inaccessible cardinal and all countable ordinals, which allows you to do all sorts of different forcings using countable collections, and then you restrict the universe to subcollections in a way consistent with measurability. The construction is complicated and unrelated to the probabilistic intuition in the main step of collapsing the inaccessible, because it is trying to stay consistent with the axiom of powerset.
The forcing freedom in set theory models still corresponds precisely to the freedom to imagine random variables take on actual new values which have not been constructed before. In this intuitive view, different from what holds in Solovay's model, the continuum hypothesis is false because, given any ordinal, I can choose random real numbers in [0,1] uniformly to inject that ordinal into R.
This intuition not only contradicts ZFC+CH, it contradicts choice, because it means the continuum is not well-orderable. But if you carry it to its logical conclusion, it contradicts even just ZF, because there should be no ordinals which cannot be injected into R. If you would like set theory to be completely consistent with the intuitions of naive probability, you should consider such a theory. But for day-to-day work, Solvay's model is sufficient. But philosophically, it is better to bite the bullet and require a set theory in which the axiom of power-set does not produce a completed infinite totality which can be viewed as a set, rather another type of object, a permanently incomplete totality, as befitting a continuum-type uncountable collection, whose elements cannot reasonably be seen as exhausted by deductions of any imagined ordinal length, and where new elements may always be forced in at will.
That such a set theory has not been constructed already is not a strike against this position, it is an invitation to construct the theory. As an added bonus, such a theory will allow different forcing universes to coexist within one larger theory of collections, countable and uncountable.