One way we formalize this "limitation" idea is via interpretative power. John Steel describes this approach carefully in several places, so you may want to read what he says, in particular at
Solomon Feferman, Harvey M. Friedman, Penelope Maddy, and John R. Steel. Does mathematics need new axioms?, The Bulletin of Symbolic Logic, 6 (4), (2000), 401-446.
Briefly, the idea is that whatever framework we choose to formalize mathematics, should strive to accommodate "the broadest possible point of view". As Steel says,
The “broadest point of view” proviso is meant to exclude from attention the temporary adoption of restrictive assumptions as a convenient device for avoiding irrelevant structure. $\mathsf{V} = \mathsf{L}$ is often assumed temporarily for such reasons by set theorists who do not believe it, just as “all functions are $C^\infty$” is sometimes assumed by differential geometers who do not believe it.
If we adopt this approach, adopting large cardinals fits within standard heuristics such as Maddy's "maximize" that suggest that our foundations should be as strong as possible. The key here is that what we are maximizing is interpretative strength. This implies maximizing consistency strength, but the goal is interpretative power rather than the other way around. Adopting large cardinals provides us with a theory within which many other theories can be appropriately interpreted, even theories that do not mention large cardinals at all. A standard example: The theory "$\mathsf{ZFC}+$ All projective sets of reals are Lebesgue measurable" can be interpreted within the theory "$\mathsf{ZFC}+$ There is an inaccessible cardinal", but not within "$\mathsf{ZFC}+$ There are no inaccessibles". If we work within a strong theory, and it turns out to be convenient for some reason to adopt (temporarily) $\mathsf{V}=\mathsf{L}$, we can do this safely, as this theory is interpretable within ours by restricting all quantifiers in the relevant statements to range over $\mathsf{L}$. On the other hand, if we adopt "$\mathsf{V}=\mathsf{L}$", we cannot faithfully interpret theories that claim the existence of measurable cardinals, for example.
Steel discusses this asymmetry carefully: The instrumentalist dodge consists in adopting, say, $\mathsf{V}=\mathsf{L}$, together with all "low level" ($\Pi^0_1$, or arithmetic, or $\Sigma^1_2$, for example) consequences of, say, "$\mathsf{ZFC}+$ There is a measurable cardinal". More generally, given a theory $T$ and a set $\Gamma$ of sentences, Steel calls $\mathrm{Inst}(T,\Gamma)$ the theory "All theorems of $T$ in $\Gamma$ are true".
There are endless variations here. The theory $\mathrm{Inst}(T, \Gamma)$, used simply as a device to avoid directly asserting $T$ while retaining all its $\Gamma$-predictions, is a mathematical parallel of the physical theory: “There are no electrons, but mid-size objects behave as if there were.” That is, it is a parallel of this theory as it might be used by a philosopher of today, not as it might have been used by a physicist in 1900. Perhaps a cleaner parallel would be the theory that the world popped into existence 5 minutes ago, looking exactly as it would if there had been a past like the one we believe in. Mis-used in this way, $\mathrm{Inst}(T, \Gamma)$ is no more than an odd way of asserting $T$; it only becomes more than that if one has a program for finding a tool for making $\Gamma$ predictions which is better than $T$, and incompatible with $T$ in the realm of non-$\Gamma$ sentences. In evaluating a retreat from $T$ to $\mathrm{Inst}(T, \Gamma)$, one should ask whether its proponent has any such competing tool in mind.
Steel's point is that there is not even a serious scenario (much less a developed tool) where assuming the negation of large cardinals (together, perhaps, with the low-level consequences of said large cardinals) ends up being advantageous in that it provides us with the means to interpret some mathematical theory that the adoption of large cardinals does not allow us to interpret.
There are a few issues here. One is that even if, theoretically, this makes sense, it may be that, in practice, the need for this all encompassing interpretability may not arise. However, there are examples where this has happened. The most prominent, that Steel mentions in his essay, is the study of regularity properties for low level projective sets, that could not really come to fruition until the theories of large cardinals and determinacy where sufficiently developed.
It doesn’t matter so much whether we can make do without this line of research; the more important question is whether we are better off with it. And it doesn't matter so much whether we call the subject which is better off (ordinary, core, normal) mathematics, or something else.
The other issue is that, perhaps, large cardinals are inconsistent after all, even if their low level consequences turn out to be consistent. There is an important point to make here: I expect that the development of inner model theory will take care of this issue if it actually arises. The tools we have developed to study large cardinals and their canonical inner models are sharp enough that we expect they'll end up uncovering inconsistencies (or mutually un-interpretable pairs of theories, or other anomalies). For example, we have developed tools to "compare" certain models via iterations. In all cases we can build these models, we can prove that the comparisons behave the way we want. In particular, this implies that theories of large cardinals have growing interpretability power matching up with their consistency strength. There are cases where we cannot yet build the models. What we do not have is cases where the models turn out to be incomparable. If we did, this would be significant. Our tools are designed in a way that I think this would be uncovered, if it actually happened. Instead, our tools keep allowing us to establish stronger and stronger positive comparability results. On the other hand, we should not discount that this optimistic reading of our partial results is what Hamkins calls "confirmation bias error", see here and here, but I think that there is really no evidence to expect the opposite, and the development of the field, and its results, suggest that we are in the presence of an authentic mathematical phenomenon. (If inner model theory keeps developing in a way that resembles current trends, and if indeed there are, say, incomparable theories, I expect our tools would reveal this by associating to the theories appropriate hod mice whose iteration strategies would end up being incomparable. On the other hand, we have theorems indicating that, so far, this does not appear to be possible; see for example Steel's preprint Normalizing iteration trees and comparing iteration strategies here.)
For more on large cardinals, why we "believe" in them, and references for much of the above, I recommend this MO thread.
Current research suggests a way in which this search for maximal interpretability power can actually reach as far as possible: This is the setting of Woodin's "Ultimate $\mathsf{L}$" theory, and its accompanying set of conjectures, notably, the $\Omega$-conjecture, that would allow us to prove that theories of large cardinals are indeed linearly ordered by interpretative power, and cofinal in the collection of theories, in terms of consistency strength. What is interesting about the modern approach is that we expect that there is not one "largest" correct theory, but rather, there may be many different theories that are mutually interpretable, and it makes no sense to prefer one to the other (on mathematical grounds), leading to a multiverse view of set theory.
That said, Woodin has suggested that, should Ultimate $\mathsf{L}$ exist and have the properties we expect, the axiom $\mathsf{V}=\mathrm{Ultimate}\mbox{-}\mathsf{L}$ plus large cardinals would provide us with a theory that in a sense is special, since it cannot be forced, and would allow us to interpret just about any other natural theory in inner models of forcing extensions of initial segments of $\mathsf{V}$. That the theory cannot be forced is partial evidence towards its "natural" rigidity, in the same sense that the theory of $\mathsf{L}$ is rigid: We have some examples of statements $\phi$ independent of $\mathsf{V}=\mathsf{L}$, but for all of them we can reasonably argue which of $\phi$ and $\lnot\phi$ is true in $\mathsf{L}$, say $\phi$, since any model of $\mathsf{V}=\mathsf{L}+\lnot\phi$ is, for instance, not well-founded.The same would be the case with $\mathsf{V}=\mathrm{Ultimate}\mbox{-}\mathsf{L}$, with the added advantage that it accomodates all large cardinals and has additional "correctness" properties. But this is all quite speculative at the moment.