Puzzle: Does the following prove that PA is inconsistent? Let PA mean the Peano axioms, let Con(PA) mean that the Peano axioms are consistent, and let Incon(PA) mean that the Peano axioms are inconsistent.
Godel's 2nd Incompleteness Theorem says that Con(PA) is not a theorem of PA, unless PA is inconsistent. Now consider the following argument that Con(PA) is a theorem of PA:
(PA and Incon(PA)) implies Con(PA), since (PA and Incon(PA)) is a contradiction, and a contradiction implies anything.
(PA and Con(PA)) implies Con(PA), obviously.
Therefore, [(PA and Incon(PA)) or (PA and Con(PA))] implies Con(PA).
Then since PA is logically equivalent to [(PA and Incon(PA)) or (PA and Con(PA))], we can conclude that PA implies Con(PA). 
Therefore, by Godel's 2nd Incompleteness Theorem, PA is inconsistent.
Where is the error? There has to be an error, or else all of mathematics is wrong.
 A: $PA+Incon(PA)$ does not prove that $Con(PA)$. What we have is that if $PA$ is inconsistent then it proves $Con(PA)$ (or that if $PA$ proves $Incon(PA)$ then it also proves $Con(PA)$). These two statements are different.
A: You're mixing up to different meanings of "inconsistent".
Let $\operatorname{Provable(X)}$ be "$X$ encodes a provable statement".
For any proposition $P$, let $\ulcorner P \urcorner$ be the encoding of $P$ as an integer. There is a theorem:
Theorem If $P$ is provable, then $PA$ implies $\operatorname{Provable(\ulcorner P \urcorner)}$
A proof sketch of this theorem is


*

*Take any proof of $P$

*Encode the proof as a natural number

*Construct that natural number in $PA$


Your mistake is assuming the converse holds as well. In particular, from $PA \text{ and } \operatorname{Provable(\operatorname{Incon}(PA))}$, one cannot conclude that $PA$ is inconsistent.
Roughly speaking, the problem is that first-order logic does not prove the infinitary theorem $x = 0 \vee x = 1 \vee x = 2 \vee \cdots $; thus, if some model of $PA$ has a number encoding a proof of $\ulcorner P \urcorner$, one cannot assume said number is actually a natural number1, which means we cannot assume that it can be decoded into a proof of $P$.
Put differently, in any model of $PA \wedge \operatorname{Incon}(PA)$, the proof of $\operatorname{Incon}(PA)$ is infinitely long, so it doesn't count as a proof. (the length of the proof is a number in the model, of course, so the model thinks its a finite number, but the model can be nonstandard and have infinitely large numbers)
1: I use "natural number" to refer specifically to the natural numbers appearing in whatever ambient theory we have developed formal logic; e.g. the model given by the decimals
A: The error is when you write

(PA and Incon(PA)) implies Con(PA), since if PA is inconsistent, PA implies that everything is both true and false.

All that (PA and Incon(PA)) implies is that PA proves Con(PA) (if PA were inconsistent, would you really believe that therefore 0=1?); and this is true internally to PA as well. That is, PA proves $$(*)\quad\mbox{If I am inconsistent, then I prove Con(PA),}$$
but PA does not prove $$(**)\quad\mbox{If I am inconsistent, then Con(PA)}.$$
That is, PA does not know that it is sound: in general, for most sentences $\varphi$, PA does not prove "If PA proves $\varphi$ then $\varphi$ is true." In fact, by Lob's theorem the only sentences for which PA proves this are the ones which PA already proves to be true!
Note that by turning your argument around, you in fact prove: 

If PA is consistent, then so is PA+Incon(PA).


Relatedly, in the same way that iterated consistency principles can be used to provide a hierarchy of extensions of PA, iterated soundness principles also provide such a hierarchy (although they are usually called reflection principles in this context). So not only does PA not prove its own soundness (even for $\Sigma^0_1$ sentences!), but it is so far from proving its own soundness that adding soundness principles to it can yield tons of additional strength.
