# Intuition behind normal subgroups

I've studied quite a bit of group theory recently, but I'm still not able to grok why normal subgroups are so important, to the extent that theorems like $(G/H)/(K/H)\approx G/K$ don't hold unless $K$ is normal, or that short exact sequences $1\to N \stackrel{f}{\to}G\stackrel{g}{\to}H\to1$ only holds when $N$ is normal.

Is there a fundamental feature of the structure of normal subgroups that makes things that only apply to normal subgroups crop up so profusely in group theory?

I'm looking here for something a bit more than "$gN=Ng$, so it acts nicely".

• Otherwise, the quotient is not a group, so most of the interesting questions do not even make sense to ask in that case. Apr 30, 2014 at 18:54
• $(G/H)/(K/H)\cong G/K$ and $N\to G\to H$ being short exact are impossible without normality.
– blue
Apr 30, 2014 at 18:54
• @TobiasKildetoft Are there any other reasons or is that pretty much it?
– Meow
Apr 30, 2014 at 19:00
• Since normal subgroups are precisely the ones that allow a natural group structure on the quotient, in some sense, it is $the$ reason. Apr 30, 2014 at 19:01
• A version of this question that I liked when I first learned abstract algebra: Why isn't a circle a ring? Why isn't $\mathbb{R}/\mathbb{Z}$ a ring? Apr 30, 2014 at 19:03

For any subgroup $H$ of $G$, you can always define an equivalence relation on $G$ given by $$g_1 \equiv g_2 \iff g_1g_2^{-1} \in H$$ This lets you define a quotient of $G$ by $H$ by looking at equivalence classes. This works perfectly well, and gives you a set of cosets, which we denote $$G/H = \{[g] = gH \mid g \in G\}$$ However, note that while we started talking about groups, we have now ended up with a set, which has less structure! (There is still some extra structure, e.g. the action of $G$ on the quotient)

We would like to define a natural group structure on this quotient, simply so that we don't end up in a completely different category. How should this new group structure behave? Well, it seems natural to ask that $$[g * h] = [g] *_{new} [h]$$ so that the map $G \to G/H$ would be a homomorphism (this is, in this context, what I mean by "natural"). So what would this mean? Let's write it out: $$(gh)H = [g * h] = [g]*_{new}[h] = (gH)(hH)$$ If you work out what these sets are, then you can see that this equation can only be true if we have that $hH = Hh$ for every $h \in G$. But this is exactly the condition that $H$ is normal.

The short answer: $H$ being normal is exactly the condition that we require so that we can put a compatible group structure on the quotient set $G/H$.

• In my opinion, this hits the nail exactly on the round flat part.
– MJD
Apr 30, 2014 at 19:30
• I prefer to think about this in the notation of modular arithmetic. We can always define an equivalence relation $\equiv$ meaning "are in the same coset" given any subgroup, and this equivalence relation is compatible with the group operation iff that subgroup is normal. Apr 30, 2014 at 20:45
• Why $(gh)H = (gH)(hH)$ can only be true if $hH = Hh$? Apr 23, 2022 at 14:56
• I know this post is several years old, but I've seen you're still active so I was hoping to ask for clarification on the final part. If I understand correctly, we wish to carry over a group structure on $G/H$ from $G$, so we consider $[g] *_{G/H}[h] = [g*_Gh] = [gh]$. For this to be well-defined we require the choices of representatives $g, h$ to be arbitrary, i.e. $[g'h'] = [gh]$ for all $g' \in [g], h'\in [h]$. From this we would eventually get $gH = Hg$ for all $g \in G$. Only then can we write $gH *_{new} hH = (gH)(hH)$. Jan 21 at 12:53
• I ask because from my point of view, the final part of the answer is in a slightly wrong order. It seems to me carrying over the group structure $[g] *_{G/H}[h] = [g*_Gh] = [gh]$ requires $gH = Hg$, from which we may define $[g]*_{new}[h] = (gH)(hH)$. Let me know what you think, apologies if I have misunderstood. Jan 21 at 12:55

A normal subgroup is a simple and unique way to characterize any homomorphism

When the words "normal subgroup" or "quotient group" are mentioned, your first reflex has to be to ask yourself "what's the associated homomorphism".

First to be fully pedantic, let's start with: why are mathematicians so obsessed with homomorphisms?

An isomorphism is a bijective function between two groups (of the same size since it's a bijection) and means that they are the exact same as far as the group structure is concerned. Very strong, and very boring.

An homomorphism however does not have to be a bijection: it can take a larger group and transform it into a smaller image group. Notably, several distinct inputs can map to the same output (non-injective).

The tradeoff is that this smaller group (the image of the homomorphism, which as shown later is isomorphic the quotient G/N) contains a "coarser" group structure than the original group, as it ignores some finer part of the original group (preview: that finer part is the normal subgroup structure). This image structure is simpler because the homomorphism can map multiple input elements to a single output element.

Mathematicians like that because breaking up bigger things into smaller things often allows to tackle the smaller parts in isolation, which often leads to simpler proofs and greater insight. It is a bit analogous to how larger integers can be broken up in to a product of their prime factors (just way more complicated largely because group multiplication is non Abelian).

More ideas on why homomorhpisms are interesting:

Now that we know why homomorphisms are interesting, let's talk about how they relate to normal subgroups

An homomorphism is a function from $$G \rightarrow H$$, and normally our intuition is that "there is a large number of possible such functions", because there are $$order(H)^{order(G)}$$ possible arbitrary functions from G to H.

However, in order to keep group structure, this is far from true, and we are much more restricted on our possible choices.

Actually, all we need to fully uniquely specify any homomorphism, is to specify its associated normal subgroup because as shown by the "Fundamental theorem on homomorphisms" mentioned in the section below:

• for any homomorphism, there is one normal group
• for any normal group, there is one homomorphism

This therefore gives a very concrete and natural way of precisely describing the homomorphism in terms of things we understand well: a subgroup of the domain. This specification also contains much less redundant information that would be present on a full "set of all input output pairs" definition.

Conversely, it also provides clear idea of what a normal group is, because homomorphisms are easy to understand (a function that respects the group operation, that's it!), and now we can understand normal groups in terms of homomorphisms.

So, how do normal groups relate to homomorphisms more precisely?

The precise way in which normal subgroups are related to homomorphisms is given in the aptly named fundamental theorem on homomorphisms. Perhaps the presentation given in the isomorphism theorem page being more understandable.

Here is a commented version of it.

Fundamental theorem on homomorphisms: Let G and H be groups, and let $$f: G \rightarrow H$$ be a homomorphism. Then:

• The kernel of f (noted ker(f)) is a normal subgroup of G

Therefore, each homomorphism uniquely specifies a normal group (the kernel of f).

• The image of f is a subgroup of H

This statement is boring. All we usually care about is the image of f, so we might as well always work with the image of f rather than this possibly larger H which contains items we know nothing about given this theorems hypothesis.

• The image of f is isomorphic to the quotient group G/ker(f)

This is kind of the converse of the first statement, and says that for every normal group (ker(f)), there is a single homomorphism (the image is isomorphic to G/ker(f)).

Remember that the quotient group is defined as the equivalence classes of:

• the normal subgroup is a class
• the cosets of the normal subgroup are the other classes

Therefore, what this part of the theorem says is that the smaller and simpler output group of the homomorphism ("the image of f") is isomorphic to the above equivalence classes.

More concretely, given a normal subgroup N, we can explicitly construct the corresponding homomorphism as:

$$f(g) = Ng$$

Here is a simplified diagram which illustrates the theorem, which tells us that every homomorphism looks like this:

where:

• f is an homomorphism from G to H

• N is a normal subgroup of G

• H = G/N is the image of f, also known as the "quotient group"

• e is the identity element of G

• $$e_H$$ is the identity element of H

• h1 and h2 arbitrarily selected non-identity elements of H

• g1 and g2 are any arbitrarily selected elements such that $$f(g1) = h1$$ and $$f(g2) = h2$$.

Since an homomorphism is not necessarily bijective, there are in general several possible choices for $$g1$$ and $$g2$$ with that property.

• f maps N to $$e_H$$, $$N \cdot g1$$ to h1 and $$N \cdot g2$$ to g2

• $$N \cdot g1$$ and $$N \cdot g2$$ are two cosets of N when it is multiplied by g1 and g2 respectively.

$$N \cdot g1$$ contains all elements $$g$$ such that $$f(g) = h1$$.

• each coset corresponds to one of the elements of H which they map to: N

• we see that N is the kernel of f by the definition of kernel, because N is the inverse image of $$e_H$$

• from this it is clear how the structure of the quotient G/N is simpler than the original G: we collapsed the structure of the entire normal group N to a single point! Therefore, an homomorphism is basically a simplification function that ignores the structure of the normal group while doing its transformation

Examples

It is now productive to try and plug some well known groups into the theorem to see what is what things look like:

Example: even/odd integers ($$C_2 = \mathbb{Z}/2\mathbb{Z}$$)

$$G = \mathbb{Z}$$ (integers)

$$H = C_2 = \{0_H, 1_H\}$$, the cyclic group of order 2, which has elements $$0_H$$ and $$1_H$$ with addition modulo 2. We add the label $$H$$ to them just to clarify that they are elements of the image $$H$$ and not of the domain $$G$$.

Then we could select either:

• $$f : \mathbb{Z} \to H, f(x) = 0_H$$ if $$x$$ is pair, $$1_H$$ if it is odd. This is an homomorphism.
• $$N = 2\mathbb{Z}$$, the set of even integers, which is a normal subgroup of $$\mathbb{Z}$$

Selecting either one of those uniquely specifies the other according to our theorem:

• if we pick $$f$$ first, then N is the preimage of the identity of H $$0_H$$, i.e. the set of all elements of G which map to $$0_H$$, which is therefore exactly the even integers

• if we pick $$N$$ first, then first necessarily all even numbers must map to $$0_H$$, because $$N$$ is the preimage of the identity of H.

Now, $$1$$ is not even, and so $$f(1)$$ has to fall in another element of H. Let's call that element $$H_1$$.

Then, $$f(3) = f(2 + 1) = f(2) + f(1) = 0_H + 1_H = 1_H$$.

If we repeat a similar reasoning for all the missing odd values, we conclude that each one of them must also fall into $$1_H$$, and so we have fully determined $$f$$.

The above shows that $$C_2 = \mathbb{Z}/2\mathbb{Z}$$.

This example also shows that in general:

• the quotient group does not need to be isomorphic to any subgroup of the domain, since $$C_2$$ is finite, and $$\mathbb{Z}$$ does not have any finite subgroups
• G does not need to be isomorphic to the direct product $$H \times N$$. For example, $$2\mathbb{Z} \times C_2$$ contains an element $$g = (0, 1)$$ such that $$g^2 = e$$ the identity, but $$\mathbb{Z}$$ contains no such element.

Other examples

By applying a similar reasoning as the above sections, we can understand the following examples which are not as verbosely worked out:

• $$\mathbb{T} = \mathbb{R}/\mathbb{Z}$$. English: the circle group (AKA 1-torus) is the quotient group of the real numbers by the integers. The underlying homomorphism $$f$$ maps each real number to their non-integer part in $$[0, 1)$$, e.g. 2.34 to 0.34 and so on. Therefore, each integer e.g. 2.00 gets mapped to the origin 0 of the image as expected. Related: What is the meaning of a torus defined by quotient group?
• for any cyclic group of order $$m$$ times $$n$$, $$C_{mn}$$, $$C_m$$ is the only subgroup of order $$m$$, and it is normal: A cyclic subgroup is normal? The associated homomorphism is the obvious "change the module" operation.

Example: the dihedral group of order 6 ($$C_2 = D_3/C_3$$)

This is the group of rotations and flips of a triangle. It has 3 rotations, with 2 flips states each, so 6 elements in total. Here's a Cayley-like graph of it:

Legend:

• r: clockwise rotation
• f: flip

This is the smallest non-Abelian group, so we might expect to see some more interesting behavior, because every Abelian group is just direct product of cyclic groups, and in $$G = H \times I$$, both $$H$$ and $$I$$ are normal in $$G$$, so everything is simple and boring.

And we do indeed see interesting behavior. There are two subgroups:

• $$C_3 = \{e, r, rr\}$$ of the unflipped rotations. It is normal.
• $$C_2 = \{e, f\}$$ of unrotated flips. It is not normal.

The homomorphism associated with the rotation normal subgroup is to map each element into either of two image elements:

• $$e_H$$: non-flipped (identity)
• $$1_H$$: flipped (the only other element)

and we just end up with a group that switches between flip and non-flip modulo 2, i.e. the cyclic group $$C_2$$.

Let's work that out more precisely. As usual, we start with the normal subgroup $$C_3$$, which is going to be the preimage of the identity.

Let's apply other transformations to it, to see that each transformation falls in either of the two image elements (flip or non-flip):

\begin{align} e C_3 &= \{e, r, rr\} \\ r C_3 &= \{r, rr, e\} = \{e, r, rr\} \\ rr C_3 &= \{rr, e, r\} = \{e, r, rr\} \\ f C_3 &= \{f, fr, frr\} \\ rf C_3 &= \{rf, rfr, rfrr\} = \{rf, f, fr\} = \{rf, f, rrf\} = \{f, rf, rrf\} \\ rrf C_3 &= \{rrf, rrfr, rrfrr\} = \{f, rf, rrf\} \\ \end{align}

and the same holds from the other side $$C_3 e$$, $$C_3 r$$, $$C_3 rr$$ and so on. So we see everything works out nicely if we take:

• $$e_H$$: image of $$\{e, r, rr\}$$
• $$1_H$$: image of $$\{f, rf, rrf\}$$

By the way, note that we can write $$G = C_2C_3$$ (the [product of two subgroups), and this type of composition is called the semi-direct product and written as $$C_3 \rtimes C_2$$.

Failing to use $$C_2$$ to generate a homomorphism in $$D_3$$

Now it is instructive to try to create a homomorphism with non-normal $$C_2$$ subgroup and see how it fails, as that gives an intuition on why we need a normal subgroup to create an homomorphism.

You will notice that it fails because the rotation on flipped elements happens in inverse direction! The inner triangle are the unflipped rotations, and the outer triangle the flipped ones. When rotations are applied, the inner and outer spin on opposite directions.

We can try that pictorially first. To start with, for sure the normal subgroup will be the kernel:

Now, we have to find a way to create two more groups of two elements that will map to the two other elements of the image.

We could be tempted to select:

but that doesn't work! E.g. if we apply:

• r to r, it goes to rr
• r to rf, it goes to f, which is in the equivalent class of the identity, so another equivalence than rr!

Therefore, we see that we can't make the homomorphism like that, as it won't be distributive over the domain's operation.

Let's try another one:

Hmmm... that one seems to work, why is that? The reason is that we had previously only checked the Cayley graph where the arrow r moves x to xr, i.e. a transformation after.

If we take the other graph, of the "transformation before" operation, then our second choice fails there because the graph is a bit different:

and so our attempt fails like the previous one:

and there is no choice that works for both.

More symbolically, if we try to create an homomorphism $$\phi$$ with kernel $$C_2 = \{e, f\}$$, we would have something like:

\begin{align} \phi(e) &= e_H \\ \phi(f) &= e_H \\ \phi(rf) &= \phi(r)\phi(f) = \phi(r)e_H = \phi(r) \\ \phi(rrf) &= \phi(fr) = \phi(f)\phi(r) = \phi(r) \\ \end{align}

Hmmm, that's bad, we already have three elements equal to $$\phi(r)$$: $$\phi(r)$$, $$\phi(rf)$$ and $$\phi(rrf)$$, but we expected 3 classes with 2 elements. And to finish things off:

$$\phi(rf) = \phi(rrf) \implies \phi(rf) = \phi(r)phi(rf) \implies \phi(r) = e_H$$

So $$r$$ must be in the kernel too besides $$\{e, f\}$$.

Continuing this we conclude that the only possible $$\phi$$ is the one that maps everything to $$e_H$$.

Why the $$gN = Ng$$ definition of a normal subgroup?

We have to think why this is a necessary and sufficient condition the relation between normal subgroups and homomorphisms to hold.

From the above discussion, we see that if there is an homomorphism, then N maps to the identity of the image ($$e_H$$).

The necessary side is therefore easy: if we have an homomorphism, because the identity commutes with anything:

$$f(gN) = f(g)f(N) = f(g)e_H \\ f(Ng) = f(N)f(g) = e_Hf(g)$$

Therefore, suppose that we take another coset like $$G1 = N \cdot g1$$, which maps to another element of H (h1).

Now for the sufficient, suppose $$gN = Ng$$. Does that imply that $$f(x) = xN$$ is an homomorphism? See e.g.: Why do we define quotient groups for normal subgroups only?

Simple group: it looks like a prime number

Now that we know all of this, it becomes clear why simple groups (a group with no Normal subgroups) are analogous to integer primes.

A group without non-trivial normal subgroups (the group itself and the identity) there is no proper homomorphism, i.e., there is no homomorphism except the trivial isomorphism and homomorphism that maps everything to the identity.

And as previously mentioned, an homomorphism breaks up the larger group into two smaller groups (N and G/N) each with part of the original structure.

Therefore simple groups are groups whose structure cannot be broken up in this way: we just can't "factor them out" with an homomorphism.

This is why so much effort was put into the classification of simple finite groups, which turned out to be such an epic result.

Quotient group: it looks like the result of a division

From the above it is also clear why the quotient group is called the "quotient group": it is because it looks similar to dividing an integer G by a factor N.

This is because much like in integer division, we produce a smaller group G/N by taking a larger group G and "dividing" it by a smaller group N.

Group extension problem: what about multiplication?

It is important to note however that this intuition that an homomorphism looks like division only works in one way: we don't really have a good multiplication analogue.

Or more precisely, we don't have a simple algorithm to solve:

Given a finite group N and a simple group S, find all groups $$G$$ such that N is a normal-subgroup of $$G$$ and G/N = S.

This happens because groups multiplication is more complex than integer multiplication (notably, non-abelian), so two groups can be composed in more complex ways than two integers, i.e. there is in general more than one possible G that solves the above for some S and N. TODO example of such a case.

You might be tempted to take the direct product of groups as a definition of multiplication, but that alone is not very satisfactory, because as mentioned at When is a group isomorphic to the product of normal subgroup and quotient group?, you would be missing out many corresponding non-trivial "divisions" (homomorphism/quotient).

A slightly better choice would actually be a semidirect product, because the direct product generates a larger group of which both smaller groups are necessarily normal (because of the trivial projection homomorphism), and the semidirect product only needs one of them to be normal. But it is still not general enough.

If we were able to do group extensions algorithmically, then we would able to classify all finite groups, because we have already classified the simple ones.

• Homomorphisms need not be surjections. Aug 19, 2020 at 9:50
• @Bungo thanks, yes! The F from Finite carried over XD Dec 27, 2020 at 10:03
• "There are $order(G) \times orger(H)$ arbitrary functions from G to H" - typo aside, pretty sure this isn't what you meant to write! Sep 24, 2021 at 21:26
• @Silverfish thanks, yes, there are way more, tried to fix it now :-) Sep 24, 2021 at 21:30
• Cheers!! Nice answer by the way! I think "what goes wrong if we tried doing things differently" is often a good approach to understanding why definitions need to be as they are for them to serve a useful purpose... Sep 24, 2021 at 21:59

The normal subgroups of $G$ are all the sets, which appear as kernel of group-homomorphisms $G \rightarrow H$.

Subgroups are the sets, which appear as images of group-homomorphism $H \rightarrow G$.

• IMHO this is the correct answer. Apr 30, 2014 at 20:45
• I also like this response. It's so simple, yet something I never knew/realized, even though I have been studying this stuff for some time. Thanks to this whole post/thread, I feel like my eyes are opened up!
– PBJ
Dec 5, 2017 at 14:52

Just to expand slightly on Simon Rose's comment

$H$ being normal is exactly the condition that we require so that we can put a compatible group structure on the quotient set $G/H$.

Suppose for each $x, y \in G$ there is $g \in G$ such that $(x H) ( y H) = g H$, that is, the product of any two left cosets of $H$ is also a left coset.

Take $x = y^{-1}$, so that $1 = y^{-1} 1 y H \in (y^{-1} H) (y H) = g H$, and thus $g H = H$. Thus for every $h \in H$ and $y \in G$ we have $$y^{-1} h y = y^{-1} h y 1 \in (y^{-1} H) ( y H) = H,$$ that is, $H$ is normal.

I was recently struck by John Baez's reply to a tweet by "Algebra Facts" about the simplicity of $$A_4$$. In it, Baez gives this informal characterization of a normal subgroup: "A normal subgroup of the symmetries of a shape is a subgroup that you can describe without pointing to any particular feature." And later in the thread, "A normal subgroup is a subgroup you can define without breaking the symmetry of the group."

I then found this correspondence Baez had with a student in 2005 in which this theme is explored and developed in more detail. In it, a (then) student by the name of Sean Fitzgerald seeks Baez's guidance in developing his intuition on this topic. Sean has some great ideas that Baez helps make a little more precise. I highly recommend it (before you read it, you should be comfortable with viewing conjugation in a group conceptually as 'changing one's point of view').

• Second link is broken :( Nov 17, 2022 at 4:54
• Here's an updated version of the second link. Dec 9, 2022 at 1:50

Once you have acquired the concept of splitting a group $$G$$ into cosets of a given $$H\le G$$, the first thing a group theorist will probably think of, is trying using the operation in $$G$$ to define an operation in the set $$G/H$$ of -say- the right cosets of $$H$$. A natural attempt is the following:

$$Ha*Hb:=Hab \tag 1$$

We are using in $$(1)$$ specific coset representatives (namely $$a$$ and $$b$$) to define an operation between cosets as such. So, we want the result of this guessed operation to be independent of these (arbitrary) choices, namely we want that:

$$a'\in Ha \wedge b'\in Hb \Longrightarrow Ha'b'=Hab \tag 2$$

Condition $$(2)$$ implies $$Ha'b'\subseteq Hab$$, namely:

\begin{alignat}{1} &\forall h,h_1,h_2\in H, \exists h'\in H\mid hh_1ah_2b=h'ab \iff \\ &\forall h,h_1,h_2\in H, \exists h'\in H\mid hh_1ah_2=h'a \iff \\ &\forall h,h_1,h_2\in H, \exists h'\in H\mid ah_2a^{-1}=(hh_1)^{-1}h' \Longrightarrow\\ &(\operatorname{take},\space e.g.,\space h=h_1=e) \space\space\forall h_2\in H, \exists h'\in H\mid ah_2a^{-1}=h' \iff \\ &aHa^{-1}\subseteq H \\ \tag 3 \end{alignat}

By the arbitrariness of $$a\in G$$, condition $$(3)$$ is precisely the normality one for $$H$$.

Therefore, the normality of $$H$$ is first and foremost a necessary condition to having, via $$(1)$$, a good definition of operation between cosets. This is seemingly even more basic than any other consideration on why matters to single out normality concept.

Edit

Out of curiosity, one might try defininig as guessed operation between cosets, in place of $$(1)$$:

$$Ha*Hb:=Ha^{-1}b \tag {1bis}$$

Good-definition constraint would lead again to the normality condition for $$H$$. The matter with $$(\operatorname{1bis})$$ is that it is associative only if $$[G:H]=2$$, so we can't seemingly get anything interesting out of it.

Exactly the same conclusion seems to hold for the other option:

$$Ha*Hb:=Hab^{-1} \tag {1ter}$$

I guess it is possible to rule out any other option than $$(m,n)=(1,1)$$ for $$Ha*Hb:=Ha^mb^n$$ defining a group operation. In that case, rather than being "natural", $$(1)$$ would simply be the only one to get a group operation between cosets by using $$G$$'s operation.

Let $$G$$ be a group. We can ask ourselves :

$$\mathbf{Q}$$) In how many ways can we put an equivalence relation $$\sim$$ on $$G$$ (ie partition set $$G$$), and put a binary operation $$\ast$$ on set $$G/{\sim}$$, such that "Equations in $$G$$ give corresponding equations between equivalence classes in $$G/{\sim}$$" that is "$$ab=c$$ in $$G$$ implies $$[a]\ast [b] = [c]$$ in $$G/{\sim}$$" ?

[Under such $$\sim, \ast$$, notice $$(G/{\sim}, \ast)$$ automatically becomes a group]

$$\underline{\textbf{Part-1}}$$ (Looking at the potential candidates for $$\sim, \ast$$)

Let $$\sim, \ast$$ be as needed.

Unwrapping the constraints one by one, $$(1) \sim \text{ is an equivalence relation }$$ and $$(2) \text{ } [a] = [a'], [b] = [b'] \implies [a]\ast [b] = [a'] \ast [b']$$ and $$(3) \text{ } [a]\ast [b] = [ab].$$

Using (3), we see (2) modifies as $$\text{ } a \sim a', b \sim b' \implies ab \sim a' b' .$$

Notice $$a \sim b \iff b^{-1} a \in [e_G]$$
($$\implies$$: As $$a \sim b$$ and $$b^{-1} \sim b ^{-1}$$, we have $$b^{-1} a \sim b ^{-1} b = e_G$$.
$$\impliedby$$: As $$b^{-1} a \sim e_G$$ and $$b\sim b$$, we have $$b(b^{-1} a) \sim b$$ ie $$a \sim b$$ )

Using this, constraint (1) can be rewritten as {$$a^{-1} a \in [e_G]$$; $$b^{-1} a \in [e_G]$$ implies $$a^{-1}b \in [e_G]$$; $$b^{-1}a , c ^{-1}b \in [e_G]$$ implies $$c^{-1} a \in [e_G]$$}, which just means "$$[e_G]$$ is a subgroup of $$G$$".
Similarly constraint (2) becomes "$$(a')^{-1} a, (b')^{-1} b \in [e_G]$$ implies $$(b')^{-1} (a')^{-1} a b \in [e_G]$$", that is "$$x, y \in [e_G]$$ implies $$(b')^{-1} (a')^{-1} (a' x) (b' y) \in [e_G]$$", that is "$$t \in [e_G]$$ implies $$g^{-1} tg \in [e_G]$$".
Constraint (3) remains the same.

So to summarize, in any such $$\sim, \ast$$, we have :

1. $$a \sim b \iff b^{-1} a \in H$$, where $$H \subseteq G$$ is a subgroup satisfying $$g^{-1} H g \subseteq H$$ for all $$g \in G$$

2. [Above condition gives $$[a] = aH$$] The operation $$*$$ satisfies $$(aH) \ast (bH) = (abH)$$

$$\underline{\textbf{Part-2}}$$ (That all such $$\sim, \ast$$ work)

Let $$H \subseteq G$$ be a subgroup with $$g^{-1} H g \subseteq H$$ for all $$g \in G$$. We can readily verify $$a \sim b \overset{\text{def}}{\iff} b^{-1} a \in H$$ and $$(aH)\ast (bH) \overset{\text{def}}{=} (abH)$$ satisfy the constraints in question.

To summarise the entire discussion, equivalence relations $$a \sim b \iff b^{-1} a \in H$$, arising from subgroups $$H$$ satisfying $$g^{-1} H g \subseteq H$$ for all $$g \in G$$, are precisely the ones we were looking for. The subgroups here are traditionally called "Normal subgroups".