A first order theory is inconsistent if it can prove every well-formed sentence. This is equivalent to being able to a sentence $A$ and it's negation $\neg A$.
The Incompleteness Theorem of Godel asserts that sufficiently strong axiom systems which include well-known theories such as Peano Arithmetics and ZFC set theory can not prove its own consistency unless it already inconsistent.
This above answers Hilbert's question : If Peano Arithemtics could prove itself to be consistent, then it was already able to prove every statement, i.e. is inconsistent. A axiom system in which everything is proveable is not very interesting.
These incompleteness result do not cause any major harm to mathematics as it is practiced. Indeed, you can not prove in Peano Arithmetics or ZFC that Peano Arithmetic or ZFC, respectively, is consistent, but, as most mathematicians do, you can just assume your axiom system is consistent (as no one has yet to find an inconsistency) and continue on your business.
So the consistency of theories is not mathematically proveable within itself. Assuming the consistency of stronger axiom systems, you can prove the consistency of the weaker. For instance assuming the consistency of ZFC, you can prove the consistency of Peano Arithmetics. Again by the incompleteness theorem, it is impossible to establish the consistency of ZFC in itself. In end, you will run into the trouble of convincing someone the stronger theories are really consistent.
Hence, you will be able to prove the consistency of your theory using method that are entirely formalizable within some first order theory. Philosophically, it is a very interesting as to whether certain axiom systems are really consistent even if you can't actually prove it formally. Some can argue that Peano arithmetic is a model of some portion of the human reasoning and is as consistent as human reasoning. Some people can informally argue that people have used the concept of arithmetics for thousands of year and applied it to myriads of real world applications without encountering any major problems. All of these are philosophical.
Ultimately, you should consider how much do you believe in arithmetics and how useful it is to you? What do you believe in more that the moon really exists or that arithmetic is consistent!