What is the formal mathematical theory behind the concept of the anti commutator used to quantize fermions? I understand Lie groups are defined by the structure constants associated with the lie brackets, which are treated as commutators in quantum mechanics, but i dont know of a math theory related to group theory to define or use an anti commutator. If Lie groups theory uses the commutator, what theory uses the anti commutator?
Finite groups (not Lie groups, which are continuous), can be specified by structure equations analogous to a Lie bracket, but more general, of which commutation or anti commutation relations are just one of infinite possibilities. Is there such variety of possible structure equations in continuous groups too?
 A: If Lie algebras are (in light of the Poincare-Birkhoff-Witt theorem) a complete axiomatization of the antisymmetric multiplication $AB-BA$ in an associative algebra, Jordan algebras are an almost-complete axiomatization of the symmetric multiplication $(AB+BA)/2$. ( http://en.wikipedia.org/wiki/Jordan_algebra ).  They are the closest thing known to an intrinsic structure related to anticommutators.
Unlike Lie algebras, there are exceptional Jordan algebras that do not come from the multiplication in associative algebras, and Jordan algebras are not known to be infinitesimal objects for groups or any other structure.  So they do not arise as often outside of (or in) quantum mechanics and the initial hopes for a Lie-like theory have not been realized.
If you mean not the anti-commutator but the supercommutator, $AB \pm BA$ with the sign depending on parity of $A$ and $B$ or their consituents (e.g., negative for bosons and positive for fermions) there is a theory of Lie superalgebras and Lie supergroups. There are new phenomena in the classification, such as continuous families of nonisomorphic simple objects.  The theory can be phrased as the "Lie group theory in the category of super-vector spaces", which in turn is a special case of Lie theory (or group theory, or algebraic geometry) in a tensor category.  So in principle there are many possible theories of commutator-like objects, but I don't know if any have been found to be interesting besides the usual theory, its super-version, and analogues in characteristic $p$.  
A: The algebraic structure corresponding most naturally to the anticommutator is that of the Clifford algebras.
One can construct a Clifford algebra by defining $n$ generating elements $\mathbf{e}_j$ with $j \in \{1,\ldots,n\}$ such that they satisfy the conditions:
$$ \mathbf{e}_i \mathbf{e}_j + \mathbf{e}_j \mathbf{e}_i = 2 \delta_{ij} \; ,$$
where $\delta_{ij}$ is the Kronecker delta. From these elements you can form products of elements of the form
$$\mathbf{e}_{i_1}\mathbf{e}_{i_2}\ldots\mathbf{e}_{i_p} \; ,$$
for $p$ varying from $0$ to $n$. There's a total of $2^n$ products that can be formed and from these, taking linear combination you can generate the rest of the algebra.
You can check that Pauli matrices and quaternions form Clifford algebras.
Now, if you take a Clifford algebra with $n$ generating elements, and construct the elements $M_{ij}=\frac{1}{2}\mathbf{e}_{i}\mathbf{e}_{j}$, you can show that they form the basis of the Lie algebra $so(n)$.
Another construct related to fermions, and which is important in the path integral representation of fermions, are the Grassmann numbers.
