Proving that the quotient of $G$ by the subgroup generated by the set of commutators is commutative. I wish to prove that $G/K(G)$ is commutative,
where $G$ is a group and $K(G)$ denotes the set $$\{aba^{-1}b^{-1}|a,b \in G\}.$$
I tried doing this using the definition (I took $X,Y$ in $K(G)$ and tried proving $XY-YX$ is in $K(G)$ but I failed...)
Edit: as pointed out by @Dylan Moreland, I meant to ask about the subgroup generated by the commutators $\{aba^{-1}b^{-1}|a,b \in G\}$.
 A: As has already been mentioned, you only get a group if you mod out by a normal subgroup.  $K(G)$ as you define it is not even a group, let alone a normal subgroup, so you need to talk about the subgroup generated by elements of the form $\{aba^{-1}b^{-1} \mid a, b \in G\}$.
When you mod out by $K(G)$, you're saying that all elements of $K(G)$ are now identified as the identity in $G / K(G)$, so it makes sense intuitively that $G / K(G)$ is commutative.  But, it's also easy to prove.  For all $a K(G), b K(G) \in G / K(G)$,
$$aba^{-1}b^{-1} K(G) = K(G),$$
so that
$$ab K(G) = ba K(G)$$
which is equivalent to
$$a K(G) \cdot b K(g) = b K(G) \cdot a K(G)$$
A: Well, the answer has already been spelt out. 
First up, I'd suggest you actually prove that the subgroup generated by $K(G)^\dagger$ is actually a normal subgroup. (Otherwise, you'd be attempting to prove a meaningless statement.)
I'll further ask you to prove a more general statement. 

Let $G$ be a group and $K$ be a normal subgroup of $G$. Prove that $G/K$ is abelian if and only if $K \supseteq \langle K(G) \rangle$ where $K(G) $ follows your definition. 

Note that the result you're after follows immediately if you take $K=\langle K(G) \rangle$

$\dagger$ The subgroup generated by $K(G)$ is called the Commutator subgroup of $G$ and is a very interesting and important subgroup of $G$. 
A: You can't, because the set of elements of the form $aba^{-1}b^{-1}$ may fail to be a subgroup; if it is not a subgroup, then you cannot take the quotient modulo $K(G)$, so you cannot even talk about $G/K(G)$.
(And $XY-YX$  makes no sense in groups; that's the commutator of two elements in a ring, not in a group)
(For examples of groups $G$ in which $K(G)$ is not a subgroup, see here, here, and here)
However, if you define $[G,G]$ to be the subgroup generated by the set $K(G)$, then it is easy to show that this subgroup is normal: if $aba^{-1}b^{-1}$ is a generator, then for every $g\in G$,
$$g^{-1}(aba^{-1}b^{-1})g = (gag^{-1})(gbg^{-1})(ga^{-1}g^{-1})(gb^{-1}g^{-1})\in K(G),$$
so the group must be normal. 
Thus, we can now meaningfully talk about $G/[G,G]$.
To show that $G/[G,G]$ is abelian, note that if $a[G,G]$ and $b[G,G]$ are elements of the quotient, then
$$(a[G,G])(b[G,G]) = ab[G,G] = ab(b^{-1}a^{-1}ba)[G,G] = ba[G,G] = (b[G,G])(a[G,G]),$$
with the middle equality because $b^{-1}a^{-1}ba\in [G,G]$. Thus, $G/[G,G]$ is abelian.
