The commutator of a group and a commutator of a ring, though similar, are fundamentally different, as you say. In each case, however, the commutator measures the "extent" to which two elements fail to commute. That is, given elements $a,b$, we wish to "compare" $ab$ and $ba$.
In the context of a group, we only have one operation: "multiplication". One way to compare two "values" using multiplication is to divide them; the "closer" you are to the identity, the closer the two values are. In particular, if $a$ and $b$ commute, we'd have
$$
\frac{ab}{ba} = 1
$$
Where $1$ denotes the group identity here.
Of course, this "division" notation is a bit ambiguous in the context of groups, since multiplication is non-commutative here. What we take to be the commutator, then, is
$$
[a,b] = (ba)^{-1}(ab) = a^{-1}b^{-1}ab
$$
With rings, it would be fantastic if we could reuse the same definition. However, the problem is that in a ring, we can't divide. Rings don't need to be groups under multiplication, they only need to be monoids (or semigroups, depending on definition).
We still, however, have an operation that allows us to "compare" $ab$ and $ba$, namely, subtraction. In particular, when $a$ and $b$ commute, we have $ab - ba = 0$. So, because it's the only way to "measure" the extent to which $a$ and $b$ commute, we define
$$
[a,b] = ab - ba
$$
in rings. It so happens that if your ring happens to allow division (that is, if you have a division ring), then it would seem that you could use both commutators. However, the group commutator works for every group, and the ring commutator works for every ring. This is why we define them the way we do.