So, as someone whose knowledge of mathematics has always come from studying it with an eye towards philosophical/foundational issues and studying it with other philosophers (who are not primarily practicing mathematicians) I am curious as to what lessons mathematicians draw from the existence of non-standard models.
Within philosophical circles it seems to be the consensus (or at least a fairly popular view) that what non-standard models show us is something about the limits of formalization. For instance, Haim Gaifman in a lecture delivered to the AMS Special Session Nonstandard Models Of Arithmetic And Set Theory (January 15-16, 2003, Baltimore, Maryland) notes the following:
If set theory is about some domain that includes uncountable sets, then any countable structure that satisfies the formalized theory must count as an unintended model. From the point of view of those who subscribe to the intended interpretation, the existence of such nonstandard models counts as a failure of the formal system to capture the semantics fully.
Now among those who subscribe to this sort of view, they tend to take the failure of categoricity in a first-order theory of Peano Arithmetic to show us that it is a second-order formulation of Peano Arithmetic which is needed. I've always taken this to result from a view that it is the semantic, rather than the syntactic, side of mathematical theorizing which holds some primacy. Often this is coupled with a view taken from Hilbert that mathematical theories (at least those that seem to have an intended interpretation) implicitly define some concept or some structure which various isomorphic models satisfy. On this sort of view, it is the concept which is of primary interest and the deductive systems are a means to discover a bit about (but generally, for reasons of incompleteness, only a bit about) what, for lack of a better word, you might call the nature of this concept.
TL;DR What lessons have mathematicians drawn from the existence of non-standard models?