If I am not wrong, many mathematicians (I believe this is not only restricted to structuralists) agree that an inconsistent formal system does not have any model. By model I mean some kind of set whose structure (objects plus their properties and relations) is represented by the formal system. But I have not seen any complaint for unsound systems. To be more specific: why are non-standard models or arithmetic considered seriously if they are unsound? How is it that people cannot imagine inconsistent structures but can imagine unsound ones. For instance, an unsound system would predict that a Turing machine will halt even when an actual one will not. Am I wrong?
One possible answer is this. Usually the reason nonstandard models of arithmetic are interesting is not that we think they really might be what we are trying to talk about when we talk about natural numbers. What's usually more interesting about them is that they show that first-order logic isn't strong enough to formulate a theory of the natural numbers which is definitely talking about the natural numbers.
The axioms of PA could be describing the natural numbers, but they could also be describing some other weird thing. We can't seem to get past this issue in first-order logic, and any attempt to "fix" the issue by adopting a stronger logic will provably run into even more weirdness. This basic conundrum is one of the major reasons that nonstandard models of arithmetic are interesting.
"My problem is that [nonstandard models] prove things that are not true about Turing machines, and they do talk about Turing machines, which are very specific objects."
One possible response begins by pointing out that a Turing machine $T$ in question is coded as a natural number, and the statement that $T$ halts is coded as a statement about natural numbers. If we plug that statement into a nonstandard model $M$ and it spits out that $T$ does not halt, we may say that the problem is that our coding of a Turing machine as a natural number, and the statement that $T$ halts as a statement about natural numbers, no longer has our intended meaning when the objects are not natural numbers but objects of $M$.
Our coding does not match our substrate (as if we were typing English on a Russian keyboard). In effect, then, we are no longer talking about Turing machines; we're talking about some other strange type of object which emerges from the interaction between our complicated arithmetical statement and our complicated nonstandard model.
For instance, it could be the case that in an arithmetically unsound system you can prove that a Turing machine that doesn't halt in the "physical" world, or rather, after a finite number of steps, would indeed halt after a transfinite number of steps (that is, for some non-standard number).
For the title question, after mulling it over, I would say that the reason we don't talk about unsound systems of deduction because we adapt the meaning of "interpretation" and "model" to fit the deductive system.
Since the other answer didn't say so explicitly or in the comments, non-standard models of arithmetic are not models of unsound systems. They are (generally) models of a consistent theory in ordinary first-order logic; in particular. They are non-standard because they are not isomorphic to the "intended" model the theory was designed for.