Intuition behind normal subgroups

Question

I've studied quite a bit of group theory recently, but I'm still not able to grok why normal subgroups are so important, to the extent that theorems like $(G/H)/(K/H)\approx G/K$ don't hold unless $K$ is normal, or that short exact sequences $1\to N \stackrel{f}{\to}G\stackrel{g}{\to}H\to1$ only holds when $N$ is normal.

Is there a fundamental feature of the structure of normal subgroups that makes things that only apply to normal subgroups crop up so profusely in group theory?

I'm looking here for something a bit more than "$gN=Ng$, so it acts nicely".

Answer 1

For any subgroup $H$ of $G$, you can always define an equivalence relation on $G$ given by $$ g_1 \equiv g_2 \iff g_1g_2^{-1} \in H $$ This lets you define a quotient of $G$ by $H$ by looking at equivalence classes. This works perfectly well, and gives you a set of cosets, which we denote $$ G/H = \{[g] = gH \mid g \in G\} $$ However, note that while we started talking about groups, we have now ended up with a set, which has less structure! (There is still some extra structure, e.g. the action of $G$ on the quotient)

We would like to define a natural group structure on this quotient, simply so that we don't end up in a completely different category. How should this new group structure behave? Well, it seems natural to ask that $$ [g * h] = [g] *_{new} [h] $$ so that the map $G \to G/H$ would be a homomorphism (this is, in this context, what I mean by "natural"). So what would this mean? Let's write it out: $$ (gh)H = [g * h] = [g]*_{new}[h] = (gH)(hH) $$ If you work out what these sets are, then you can see that this equation can only be true if we have that $hH = Hh$ for every $h \in G$. But this is exactly the condition that $H$ is normal.

The short answer: $H$ being normal is exactly the condition that we require so that we can put a compatible group structure on the quotient set $G/H$.

Answer 2

A normal subgroup is a simple and unique way to characterize any homomorphism

When the words "normal subgroup" or "quotient group" are mentioned, your first reflex has to be to ask yourself "what's the associated homomorphism".

First to be fully pedantic, let's start with: why are mathematicians so obsessed with homomorphisms?

An isomorphism is a bijective function between two groups (of the same size since it's a bijection) and means that they are the exact same as far as the group structure is concerned. Very strong, and very boring.

An homomorphism however does not have to be a bijection: it can take a larger group and transform it into a smaller image group. Notably, several distinct inputs can map to the same output (non-injective).

The tradeoff is that this smaller group (the image of the homomorphism, which as shown later is isomorphic the quotient G/N) contains a "coarser" group structure than the original group, as it ignores some finer part of the original group (preview: that finer part is the normal subgroup structure). This image structure is simpler because the homomorphism can map multiple input elements to a single output element.

Mathematicians like that because breaking up bigger things into smaller things often allows to tackle the smaller parts in isolation, which often leads to simpler proofs and greater insight. It is a bit analogous to how larger integers can be broken up in to a product of their prime factors (just way more complicated largely because group multiplication is non Abelian).

More ideas on why homomorhpisms are interesting:

• Why the term and the concept of quotient group?
• they show up in Lie Groups. This wiki page actually mentions that some of the isomorphism theorems were actually first formulated by Noether, which makes sense since Lie groups are key to Noether's theorem. The definition of group representations involves an homomorphism.
• the unsolvability of the quintic equation is linked to the simplicity of $A_5$. As mentioned below, "simple" means "without a normal subgroup", or more intuitively "without non-trivial homomorphisms"

Now that we know why homomorphisms are interesting, let's talk about how they relate to normal subgroups

An homomorphism is a function from $G \rightarrow H$, and normally our intuition is that "there is a large number of possible such functions", because there are $order(H)^{order(G)}$ possible arbitrary functions from G to H.

However, in order to keep group structure, this is far from true, and we are much more restricted on our possible choices.

Actually, all we need to fully uniquely specify any homomorphism, is to specify its associated normal subgroup because as shown by the "Fundamental theorem on homomorphisms" mentioned in the section below:

• for any homomorphism, there is one normal group
• for any normal group, there is one homomorphism

This therefore gives a very concrete and natural way of precisely describing the homomorphism in terms of things we understand well: a subgroup of the domain. This specification also contains much less redundant information that would be present on a full "set of all input output pairs" definition.

Conversely, it also provides clear idea of what a normal group is, because homomorphisms are easy to understand (a function that respects the group operation, that's it!), and now we can understand normal groups in terms of homomorphisms.

So, how do normal groups relate to homomorphisms more precisely?

The precise way in which normal subgroups are related to homomorphisms is given in the aptly named fundamental theorem on homomorphisms. Perhaps the presentation given in the isomorphism theorem page being more understandable.

Here is a commented version of it.

Fundamental theorem on homomorphisms: Let G and H be groups, and let $f: G \rightarrow H$ be a homomorphism. Then:

• The kernel of f (noted ker(f)) is a normal subgroup of G

  Therefore, each homomorphism uniquely specifies a normal group (the kernel of f).

• The image of f is a subgroup of H

  This statement is boring. All we usually care about is the image of f, so we might as well always work with the image of f rather than this possibly larger H which contains items we know nothing about given this theorems hypothesis.

• The image of f is isomorphic to the quotient group G/ker(f)

  This is kind of the converse of the first statement, and says that for every normal group (ker(f)), there is a single homomorphism (the image is isomorphic to G/ker(f)).

  Remember that the quotient group is defined as the equivalence classes of:

  • the normal subgroup is a class
  • the cosets of the normal subgroup are the other classes

  Therefore, what this part of the theorem says is that the smaller and simpler output group of the homomorphism ("the image of f") is isomorphic to the above equivalence classes.

  More concretely, given a normal subgroup N, we can explicitly construct the corresponding homomorphism as:

  $$f(g) = Ng$$

Here is a simplified diagram which illustrates the theorem, which tells us that every homomorphism looks like this:

where:

• f is an homomorphism from G to H

• N is a normal subgroup of G

• H = G/N is the image of f, also known as the "quotient group"

• e is the identity element of G

• $e_H$ is the identity element of H

• h1 and h2 arbitrarily selected non-identity elements of H

• g1 and g2 are any arbitrarily selected elements such that $f(g1) = h1$ and $f(g2) = h2$.

  Since an homomorphism is not necessarily bijective, there are in general several possible choices for $g1$ and $g2$ with that property.

• f maps N to $e_H$, $N \cdot g1$ to h1 and $N \cdot g2$ to g2

• $N \cdot g1$ and $N \cdot g2$ are two cosets of N when it is multiplied by g1 and g2 respectively.

  $N \cdot g1$ contains all elements $g$ such that $f(g) = h1$.

• each coset corresponds to one of the elements of H which they map to: N

• we see that N is the kernel of f by the definition of kernel, because N is the inverse image of $e_H$

• from this it is clear how the structure of the quotient G/N is simpler than the original G: we collapsed the structure of the entire normal group N to a single point! Therefore, an homomorphism is basically a simplification function that ignores the structure of the normal group while doing its transformation

Examples

It is now productive to try and plug some well known groups into the theorem to see what is what things look like:

Example: even/odd integers ($C_2 = \mathbb{Z}/2\mathbb{Z}$)

$G = \mathbb{Z}$ (integers)

$H = C_2 = \{0_H, 1_H\}$, the cyclic group of order 2, which has elements $0_H$ and $1_H$ with addition modulo 2. We add the label $H$ to them just to clarify that they are elements of the image $H$ and not of the domain $G$.

Then we could select either:

• $f : \mathbb{Z} \to H, f(x) = 0_H$ if $x$ is pair, $1_H$ if it is odd. This is an homomorphism.
• $N = 2\mathbb{Z}$, the set of even integers, which is a normal subgroup of $\mathbb{Z}$

Selecting either one of those uniquely specifies the other according to our theorem:

• if we pick $f$ first, then N is the preimage of the identity of H $0_H$, i.e. the set of all elements of G which map to $0_H$, which is therefore exactly the even integers

• if we pick $N$ first, then first necessarily all even numbers must map to $0_H$, because $N$ is the preimage of the identity of H.

  Now, $1$ is not even, and so $f(1)$ has to fall in another element of H. Let's call that element $H_1$.

  Then, $f(3) = f(2 + 1) = f(2) + f(1) = 0_H + 1_H = 1_H$.

  If we repeat a similar reasoning for all the missing odd values, we conclude that each one of them must also fall into $1_H$, and so we have fully determined $f$.

The above shows that $C_2 = \mathbb{Z}/2\mathbb{Z}$.

This example also shows that in general:

• the quotient group does not need to be isomorphic to any subgroup of the domain, since $C_2$ is finite, and $\mathbb{Z}$ does not have any finite subgroups
• G does not need to be isomorphic to the direct product $H \times N$. For example, $2\mathbb{Z} \times C_2$ contains an element $g = (0, 1)$ such that $g^2 = e$ the identity, but $\mathbb{Z}$ contains no such element.

Other examples

By applying a similar reasoning as the above sections, we can understand the following examples which are not as verbosely worked out:

• $\mathbb{T} = \mathbb{R}/\mathbb{Z}$. English: the circle group (AKA 1-torus) is the quotient group of the real numbers by the integers. The underlying homomorphism $f$ maps each real number to their non-integer part in $[0, 1)$, e.g. 2.34 to 0.34 and so on. Therefore, each integer e.g. 2.00 gets mapped to the origin 0 of the image as expected. Related: What is the meaning of a torus defined by quotient group?
• for any cyclic group of order $m$ times $n$, $C_{mn}$, $C_m$ is the only subgroup of order $m$, and it is normal: A cyclic subgroup is normal? The associated homomorphism is the obvious "change the module" operation.

Example: the dihedral group of order 6 ($C_2 = D_3/C_3$)

This is the group of rotations and flips of a triangle. It has 3 rotations, with 2 flips states each, so 6 elements in total. Here's a Cayley-like graph of it:

SVG source.

Legend:

• r: clockwise rotation
• f: flip

This is the smallest non-Abelian group, so we might expect to see some more interesting behavior, because every Abelian group is just direct product of cyclic groups, and in $G = H \times I$, both $H$ and $I$ are normal in $G$, so everything is simple and boring.

And we do indeed see interesting behavior. There are two subgroups:

• $C_3 = \{e, r, rr\}$ of the unflipped rotations. It is normal.
• $C_2 = \{e, f\}$ of unrotated flips. It is not normal.

The homomorphism associated with the rotation normal subgroup is to map each element into either of two image elements:

• $e_H$: non-flipped (identity)
• $1_H$: flipped (the only other element)

and we just end up with a group that switches between flip and non-flip modulo 2, i.e. the cyclic group $C_2$.

Let's work that out more precisely. As usual, we start with the normal subgroup $C_3$, which is going to be the preimage of the identity.

Let's apply other transformations to it, to see that each transformation falls in either of the two image elements (flip or non-flip):

$$ \begin{align} e C_3 &= \{e, r, rr\} \\ r C_3 &= \{r, rr, e\} = \{e, r, rr\} \\ rr C_3 &= \{rr, e, r\} = \{e, r, rr\} \\ f C_3 &= \{f, fr, frr\} \\ rf C_3 &= \{rf, rfr, rfrr\} = \{rf, f, fr\} = \{rf, f, rrf\} = \{f, rf, rrf\} \\ rrf C_3 &= \{rrf, rrfr, rrfrr\} = \{f, rf, rrf\} \\ \end{align} $$

and the same holds from the other side $C_3 e$, $C_3 r$, $C_3 rr$ and so on. So we see everything works out nicely if we take:

• $e_H$: image of $\{e, r, rr\}$
• $1_H$: image of $\{f, rf, rrf\}$

By the way, note that we can write $G = C_2C_3$ (the [product of two subgroups), and this type of composition is called the semi-direct product and written as $C_3 \rtimes C_2$.

Failing to use $C_2$ to generate a homomorphism in $D_3$

Now it is instructive to try to create a homomorphism with non-normal $C_2$ subgroup and see how it fails, as that gives an intuition on why we need a normal subgroup to create an homomorphism.

You will notice that it fails because the rotation on flipped elements happens in inverse direction! The inner triangle are the unflipped rotations, and the outer triangle the flipped ones. When rotations are applied, the inner and outer spin on opposite directions.

We can try that pictorially first. To start with, for sure the normal subgroup will be the kernel:

Now, we have to find a way to create two more groups of two elements that will map to the two other elements of the image.

We could be tempted to select:

but that doesn't work! E.g. if we apply:

• r to r, it goes to rr
• r to rf, it goes to f, which is in the equivalent class of the identity, so another equivalence than rr!

Therefore, we see that we can't make the homomorphism like that, as it won't be distributive over the domain's operation.

Let's try another one:

Hmmm... that one seems to work, why is that? The reason is that we had previously only checked the Cayley graph where the arrow r moves x to xr, i.e. a transformation after.

If we take the other graph, of the "transformation before" operation, then our second choice fails there because the graph is a bit different:

and so our attempt fails like the previous one:

and there is no choice that works for both.

More symbolically, if we try to create an homomorphism $\phi$ with kernel $C_2 = \{e, f\}$, we would have something like:

$$ \begin{align} \phi(e) &= e_H \\ \phi(f) &= e_H \\ \phi(rf) &= \phi(r)\phi(f) = \phi(r)e_H = \phi(r) \\ \phi(rrf) &= \phi(fr) = \phi(f)\phi(r) = \phi(r) \\ \end{align} $$

Hmmm, that's bad, we already have three elements equal to $\phi(r)$: $\phi(r)$, $\phi(rf)$ and $\phi(rrf)$, but we expected 3 classes with 2 elements. And to finish things off:

$$\phi(rf) = \phi(rrf) \implies \phi(rf) = \phi(r)phi(rf) \implies \phi(r) = e_H$$

So $r$ must be in the kernel too besides $\{e, f\}$.

Continuing this we conclude that the only possible $\phi$ is the one that maps everything to $e_H$.

Why the $gN = Ng$ definition of a normal subgroup?

We have to think why this is a necessary and sufficient condition the relation between normal subgroups and homomorphisms to hold.

From the above discussion, we see that if there is an homomorphism, then N maps to the identity of the image ($e_H$).

The necessary side is therefore easy: if we have an homomorphism, because the identity commutes with anything:

$$ f(gN) = f(g)f(N) = f(g)e_H \\ f(Ng) = f(N)f(g) = e_Hf(g) $$

Therefore, suppose that we take another coset like $G1 = N \cdot g1$, which maps to another element of H (h1).

Now for the sufficient, suppose $gN = Ng$. Does that imply that $f(x) = xN$ is an homomorphism? See e.g.: Why do we define quotient groups for normal subgroups only?

Simple group: it looks like a prime number

Now that we know all of this, it becomes clear why simple groups (a group with no Normal subgroups) are analogous to integer primes.

A group without non-trivial normal subgroups (the group itself and the identity) there is no proper homomorphism, i.e., there is no homomorphism except the trivial isomorphism and homomorphism that maps everything to the identity.

And as previously mentioned, an homomorphism breaks up the larger group into two smaller groups (N and G/N) each with part of the original structure.

Therefore simple groups are groups whose structure cannot be broken up in this way: we just can't "factor them out" with an homomorphism.

This is why so much effort was put into the classification of simple finite groups, which turned out to be such an epic result.

Quotient group: it looks like the result of a division

From the above it is also clear why the quotient group is called the "quotient group": it is because it looks similar to dividing an integer G by a factor N.

This is because much like in integer division, we produce a smaller group G/N by taking a larger group G and "dividing" it by a smaller group N.

See also: Why the term and the concept of quotient group?

Group extension problem: what about multiplication?

It is important to note however that this intuition that an homomorphism looks like division only works in one way: we don't really have a good multiplication analogue.

Or more precisely, we don't have a simple algorithm to solve:

Given a finite group N and a simple group S, find all groups $G$ such that N is a normal-subgroup of $G$ and G/N = S.

This happens because groups multiplication is more complex than integer multiplication (notably, non-abelian), so two groups can be composed in more complex ways than two integers, i.e. there is in general more than one possible G that solves the above for some S and N. TODO example of such a case.

You might be tempted to take the direct product of groups as a definition of multiplication, but that alone is not very satisfactory, because as mentioned at When is a group isomorphic to the product of normal subgroup and quotient group?, you would be missing out many corresponding non-trivial "divisions" (homomorphism/quotient).

A slightly better choice would actually be a semidirect product, because the direct product generates a larger group of which both smaller groups are necessarily normal (because of the trivial projection homomorphism), and the semidirect product only needs one of them to be normal. But it is still not general enough.

If we were able to do group extensions algorithmically, then we would able to classify all finite groups, because we have already classified the simple ones.

See also: How is a group made up of simple groups?

Answer 3

The normal subgroups of $G$ are all the sets, which appear as kernel of group-homomorphisms $G \rightarrow H$.

Subgroups are the sets, which appear as images of group-homomorphism $H \rightarrow G$.

Answer 4

Just to expand slightly on Simon Rose's comment

$H$ being normal is exactly the condition that we require so that we can put a compatible group structure on the quotient set $G/H$.

Suppose for each $x, y \in G$ there is $g \in G$ such that $(x H) ( y H) = g H$, that is, the product of any two left cosets of $H$ is also a left coset.

Take $x = y^{-1}$, so that $1 = y^{-1} 1 y H \in (y^{-1} H) (y H) = g H$, and thus $g H = H$. Thus for every $h \in H$ and $y \in G$ we have $$ y^{-1} h y = y^{-1} h y 1 \in (y^{-1} H) ( y H) = H, $$ that is, $H$ is normal.

Answer 5

I was recently struck by John Baez's reply to a tweet by "Algebra Facts" about the simplicity of $A_4$. In it, Baez gives this informal characterization of a normal subgroup: "A normal subgroup of the symmetries of a shape is a subgroup that you can describe without pointing to any particular feature." And later in the thread, "A normal subgroup is a subgroup you can define without breaking the symmetry of the group."

I then found this correspondence Baez had with a student in 2005 in which this theme is explored and developed in more detail. In it, a (then) student by the name of Sean Fitzgerald seeks Baez's guidance in developing his intuition on this topic. Sean has some great ideas that Baez helps make a little more precise. I highly recommend it (before you read it, you should be comfortable with viewing conjugation in a group conceptually as 'changing one's point of view').

Answer 6

Once you have acquired the concept of splitting a group $G$ into cosets of a given $H\le G$, the first thing a group theorist will probably think of, is trying using the operation in $G$ to define an operation in the set $G/H$ of -say- the right cosets of $H$. A natural attempt is the following:

$$Ha*Hb:=Hab \tag 1$$

We are using in $(1)$ specific coset representatives (namely $a$ and $b$) to define an operation between cosets as such. So, we want the result of this guessed operation to be independent of these (arbitrary) choices, namely we want that:

$$a'\in Ha \wedge b'\in Hb \Longrightarrow Ha'b'=Hab \tag 2$$

Condition $(2)$ implies $Ha'b'\subseteq Hab$, namely:

\begin{alignat}{1} &\forall h,h_1,h_2\in H, \exists h'\in H\mid hh_1ah_2b=h'ab \iff \\ &\forall h,h_1,h_2\in H, \exists h'\in H\mid hh_1ah_2=h'a \iff \\ &\forall h,h_1,h_2\in H, \exists h'\in H\mid ah_2a^{-1}=(hh_1)^{-1}h' \Longrightarrow\\ &(\operatorname{take},\space e.g.,\space h=h_1=e) \space\space\forall h_2\in H, \exists h'\in H\mid ah_2a^{-1}=h' \iff \\ &aHa^{-1}\subseteq H \\ \tag 3 \end{alignat}

By the arbitrariness of $a\in G$, condition $(3)$ is precisely the normality one for $H$.

Therefore, the normality of $H$ is first and foremost a necessary condition to having, via $(1)$, a good definition of operation between cosets. This is seemingly even more basic than any other consideration on why matters to single out normality concept.

---

Edit

Out of curiosity, one might try defininig as guessed operation between cosets, in place of $(1)$:

$$Ha*Hb:=Ha^{-1}b \tag {1bis}$$

Good-definition constraint would lead again to the normality condition for $H$. The matter with $(\operatorname{1bis})$ is that it is associative only if $[G:H]=2$, so we can't seemingly get anything interesting out of it.

Exactly the same conclusion seems to hold for the other option:

$$Ha*Hb:=Hab^{-1} \tag {1ter}$$

I guess it is possible to rule out any other option than $(m,n)=(1,1)$ for $Ha*Hb:=Ha^mb^n$ defining a group operation. In that case, rather than being "natural", $(1)$ would simply be the only one to get a group operation between cosets by using $G$'s operation.

Answer 7

Given a group $G$ and a normal subgroup $N$, when we take the quotient $G/N$ we're considering what $G$ would look like if every element of $N$ was equal to $0$. Take for example $\mathbb{Z}/N$, where $N=3\mathbb{Z}$. This asks the question "How would the integers be different if every multiple of $3$ were equal to 0?" If this were the case, every multiple of $3$ would equal every other multiple of $3$ -- they would all equal $0$. So we may as well refer to them as a whole, as $N$, i.e. to say things like $N=0$ or $N+N=N$, where we now think of $N$ as behaving like the single element $0$. If every multiple of $3$ equals $0$, we can then study what the other elements of $\mathbb{Z}/N$ look like. We know that $1+0=1$, so it should be true that $1+N=1$. In other words, the elements $1+3k$, for $k\in \mathbb{Z}$, are all equal to $1$ in $\mathbb{Z}/N$. This is because since adding anything equal to $0$ in $G/N$ shouldn't change the value of $1$, and each $3k$ above is in $N$ and thus equal to $0$. The same goes for $2$ -- since $2+0=2$, we see that $2+N=2$ and thus $2+3k=2$ for each $k\in\mathbb{Z}$. You can check that this provides a complete description of the elements of $\mathbb{Z}/N$!

The upshot of this line of reasoning is that the quotient group $G/N$ is attained by treating $N$ collectively as the $0$ element. This lets us provide some intuition behind your original question: $0$ commutes with every element of a given group, even if the group itself is not commutative. Thus, if we treat $N$ as the $0$ element, it had better commute with every other element. This is the condition that for all $g\in G$, $g+N=N+g$. It turns out that this is all we need to define an addition operation in $G/N$ based on the addition operation in $G$: if $g_1+N$ and $g_2+N$ are two cosets in $G/N$, their sum is $g_1+N+g_2+N=g_1+g_2+N+N=g_1+g_2+N$. This only requires that $g_2+N=N+g_2$, or in other words, that $N$ commutes with $g_2$ as a set. Each element of $N$ doesn't need to commute with $g_2$, but in total, it must be the case that the set $g_2+N$ equals the set $N+g_2$. Thus, the normality condition intuitively expresses the fact that $N$ commutes as a set with each element $g\in G$, and hence as a set behaves sufficiently like the $0$ element to be identified as $0$ in the quotient group $G/N$. We can use this to justify that $N$ ought to be a group as well. We know that $0+0=0$, so $N+N=N$ and we get that $N$ is closed; $0$ is its own additive inverse, so $N$ must be closed under inverses; $N$ inherits associativity from its ambient group $G$; finally, of course,$0$ must be in $N$ if $N$ is the collection of elements which "behave like $0$" and surely this notion can't exclude the original $0$ element. (If we think of the group operation of $G$ written as multiplication, we get the form $gN=Ng$ that you write in your question and the rest of these explanations translate nicely).

As a bonus, this "behaving like $0$" reasoning extends to let us understand the conditions for ideals of rings intuitively as well. Consider a ring $R$ and a quotient $R/I$ by an ideal $I$. In the case of rings, we have two operations to respect in the quotient structure. Acting as $0$ in addition amounts to requiring that an ideal $I$ be a normal subgroup, for the same reasons as above. What's left is to consider what $0$ does with respect to multiplication: the only ring axiom relating $0$ to multiplication is the distributive property, which allows us to conclude that the product $0\cdot r=0$ for any $r\in R$. Translated to the context of ideals, if $I$ is behaving as $0$ does, it had better be the case that for any $r\in R$, $r\cdot I=I$ (where we replace $0$ on both sides by I), justifying the requirement that ideals absorb products.

There are a few details that we would need to add in to make this explanation complete, but hopefully this provides a clear intuitive explanation!

Answer 8

Let $G$ be a group. We can ask ourselves :

$\mathbf{Q}$) In how many ways can we put an equivalence relation $\sim$ on $G$ (ie partition set $G$), and put a binary operation $\ast$ on set $G/{\sim}$, such that "Equations in $G$ give corresponding equations between equivalence classes in $G/{\sim}$" that is "$ab=c$ in $G$ implies $[a]\ast [b] = [c]$ in $G/{\sim}$" ?

[Under such $\sim, \ast$, notice $(G/{\sim}, \ast)$ automatically becomes a group]

$\underline{\textbf{Part-1}}$ (Looking at the potential candidates for $ \sim, \ast$)

Let $\sim, \ast$ be as needed.

Unwrapping the constraints one by one, $$(1) \sim \text{ is an equivalence relation } $$ and $$ (2) \text{ } [a] = [a'], [b] = [b'] \implies [a]\ast [b] = [a'] \ast [b'] $$ and $$(3) \text{ } [a]\ast [b] = [ab].$$

Using (3), we see (2) modifies as $\text{ } a \sim a', b \sim b' \implies ab \sim a' b' .$

Notice $a \sim b \iff b^{-1} a \in [e_G]$
($\implies$: As $a \sim b$ and $b^{-1} \sim b ^{-1}$, we have $b^{-1} a \sim b ^{-1} b = e_G$.
$\impliedby$: As $b^{-1} a \sim e_G$ and $b\sim b$, we have $b(b^{-1} a) \sim b$ ie $a \sim b$ )

Using this, constraint (1) can be rewritten as {$a^{-1} a \in [e_G]$; $b^{-1} a \in [e_G]$ implies $a^{-1}b \in [e_G]$; $b^{-1}a , c ^{-1}b \in [e_G]$ implies $c^{-1} a \in [e_G]$}, which just means "$[e_G]$ is a subgroup of $G$".
Similarly constraint (2) becomes "$(a')^{-1} a, (b')^{-1} b \in [e_G]$ implies $(b')^{-1} (a')^{-1} a b \in [e_G]$", that is "$x, y \in [e_G]$ implies $(b')^{-1} (a')^{-1} (a' x) (b' y) \in [e_G]$", that is "$t \in [e_G]$ implies $g^{-1} tg \in [e_G]$".
Constraint (3) remains the same.

So to summarize, in any such $\sim, \ast$, we have :

1. $ a \sim b \iff b^{-1} a \in H $, where $ H \subseteq G $ is a subgroup satisfying $ g^{-1} H g \subseteq H $ for all $ g \in G $

2. [Above condition gives $[a] = aH$] The operation $ * $ satisfies $(aH) \ast (bH) = (abH)$

$\underline{\textbf{Part-2}}$ (That all such $ \sim, \ast $ work)

Let $ H \subseteq G $ be a subgroup with $ g^{-1} H g \subseteq H $ for all $ g \in G $. We can readily verify $ a \sim b \overset{\text{def}}{\iff} b^{-1} a \in H $ and $ (aH)\ast (bH) \overset{\text{def}}{=} (abH) $ satisfy the constraints in question.

---

To summarise the entire discussion, equivalence relations $ a \sim b \iff b^{-1} a \in H $, arising from subgroups $ H $ satisfying $ g^{-1} H g \subseteq H$ for all $ g \in G$, are precisely the ones we were looking for. The subgroups here are traditionally called "Normal subgroups".