I was studying the Foundations of Mathematics and a question was bugging me for quite some while. Does our choice of axioms depend upon the system we are studying? Could it be that if some statement is an axiom in one system it is a theorem in another? For example, I was learning Peano Axioms where we prove theorems establishing commutativity, associativity, distributivity etc. of number systems (natural, integers, rational, real, etc.) and then when I began Abstract Algebra, it treated these propositions as axioms. Even in Elementary Algebra these properties of natural or real numbers are taken to be demonstrable axioms. So is our choice of axioms subjective? It would be great if someone could shed some light on this.
Yes, the choice of axioms is subjective.
There can be multiple sets of axioms describing the same system, and since the systems are the same, you can prove the same statements in them and thus, choosing a set of axioms $A$ and $B$ to describe one and the same system, one can prove the statements in $A$ from $B$ and vice versa. So yes, the choice of axioms depend on the system - but there can be multiple choices of axioms describing the same system.
Note however that taking intuitive and common axioms for existing systems makes your proofs more readable and easier to understand, and as such, you probably don't want to choose your axioms too out-of-the-ordinary even though they describe the same system. You could, but you probably shouldn't.
There was a fashion once, when mathematics was first being formalised, to find the weakest set of statements which would serve as axioms. For example, taking a one-sided identity and a one sided inverse as axioms of group theory (which works if you get the sides right, and have associativity) rather than having two-sided identity and inverse as axioms. Then you prove the two-sided version, and then forget about it and work as if it were two-sided all the time.
Now that kind of thing is of interest, but not so much to people who are doing group theory. For that you want axioms which are easy to work with.
Group theory also helps us to understand the need for particular axioms - the existence of non-commutative groups tells us that we cannot take commutativity for granted. It may follow from other axioms in general or in particular cases (every group of order $4$ is commutative, for example). But if you want it, it is convenient to state it. Whether it matters if it is redundant depends on your point of view.
It is possible to be a bit silly in the other direction, and simply take every true statement about a system as an axiom. There may then be problems identifying which statements are axioms, and there is no much point trying to prove anything.
As a further example take multiplication. In Peano arithmetic this has to be defined, and its properties inferred from the axioms. If you want your algebraic system to accommodate multiplication - perhaps because of an application you have in mind, there is another way to go, which may be more convenient - take the existence and properties of products as axioms. Either works, but which is better depends on your point of view and purpose.
Then, with all sets of axioms there is the question of whether or not they are consistent (or occasionally apparently good axioms define only trivial objects of no real interest). Consistency can be a deep property, but simple axioms clearly stated can help to spot some of the more obvious possible inconsistencies.
Another question is whether, for example, there is a model of Peano Arithmetic within some formulation of Set Theory. i.e. whether one axiomatic system "contains" another or might act as a foundation for it. So some mathematicians are interested in providing structures (like Set Theory) in which mathematics can be done.
Yes, a body of mathematical propositions can be deduced from many sets of primitive propositions (referred to as Pps hereafter), and it is possible that Pps in one deductive system are theorems in another.
Evaluating the merits of one set of Pps over another constitutes the bulk of mathematical philosophy. If you read Whitehead & Russell's Principia Mathematica you will frequently come across discussions about why the authors think such-a-such definition or theorem is more fundamental than another.
The guiding principles of choosing one set over another can be summarized as follows:
Simplicity or Occam's razor. A smaller set of Pps is preferable than a larger set. If two sets have the same number of Pps, which one is preferable is up to the author, but the authors always have a reason - this is characteristic of philosophical work.
Self-evidence. Each Pp should be as luminously self-evident as possible. Take Principia Mathematica for example.
✳1.1 Anything implied by a true proposition is true.
✳1.2 $\vdash :p\vee p. \supset .p$
which states that "p or p implies p."
To demonstrate a point. Whitehead & Russell's Pps are logical propositions; by deducing mathematics from logical principles W&R demonstrated that logic and mathematics are one and the same.
Note: I somewhat felt bludgeoned by the careless use of such words as describe and define in some of the other answers. A set of Pps implies or determines a body of mathematical propositions, but it neither defines nor describes such a body. Imply, define and describe have precise, distinct meanings, the use which are fundamental in mathematical philosophy.
The field you are talking about is call Reverse Mathematics, and there is a number of related questions under the tag reverse-math. A typical procedure would be, instead of using the usual axioms to prove, say, the extremal value theorem (EVT), postulating the EVT to see what former "axioms" can be deduced from it or are equivalent to it. Here a certain bare minimum is assumed which is usually denoted RCA$_0$; see reverse mathematics.