What is the meaning of the parentheses in $\phi^{-1}\left[\{\phi(g)\}\right]=gH=Hg$? I am studying homomorphisms is groups and i saw a theorem saying:
For $g$ in a group $G$, the cosets $gH$ and $Hg$ are the same, and collapsed onto the single element $\phi(g)$ by $\phi$. That is, $\phi^{-1}\left[\{\phi(g)\}\right]=gH=Hg$.
I am confused about the parentheses here. I think $\{\phi(g)\}$ denotes the set, formed by the elements of the mapping, but i am not sure what $[\ ]$ represent. Can anyone tell how should i interpret those parentheses? Or explain the theorem in other words?
Thank you
 A: $$Φ^{-1}\left[\{Φ(g)\}\right]=gH=Hg\tag{1}$$
$(1)$ simply denotes the preimage (the inverse mapping) of the image of the "set": $\{\Phi(g)\} \subseteq G/H$ under $\Phi$, assuming in this case we have that $H$ is normal. Since the "object" being mapped by the homomorphism $\Phi$ is technically a set, instead of enclosing the set in parentheses, as we would for an "element", it's using square brackets, but it would mean the same to write $Φ^{-1}\left(\{Φ(g)\}\right)=gH=Hg$
Added: I agree with Ted Shifrin that your text is being excessively pedantic (pedantic: overly concerned with formal rules and trivial points of learning!)
We more typically refer $\Phi (g)$ as an element in $G/H$, and could express the preimage of an element $\Phi(g) \in H/G$ by simply using:  $$Φ^{-1}[Φ(g)]=gH=Hg \;\;\;\text{ or}\;\;\; \Phi^{-1}\left(\Phi(g)\right) = gH = Hg\tag{2}$$
A: Some use the notation $f^{-1}[X]$ to denote the preimage of a set $X$ under a map $f:A\to B\supseteq X$: apparently the brackets prevent ambiguity in set theory, as one of my instructors explained to me.
Here is an example justifying the need. Suppose we build the natural numbers $\bf N$ as ordinals:


*

*$0:=\{\}$

*$1:=\{0\}=\{\{\}\}$

*$2:=\{0,1\}=\{\{\},\{\{\}\}\}$

*$3:=\{0,1,2\}=\{\{\},\{\{\}\},\{\{\},\{\{\}\}\}\}$

*$4:=\{0,1,2,3\}=\cdots$

*$\cdots$

*$n:=\{{\rm ordinals}<n\}$

*$\cdots$


Now let $f:\bf N\to N$ be an injective function. Suppose $f(0)=1$ and $f(1)=0$. Then the inverse function $f^{-1}$ applied to $0$ is $1$, and applied to $1$ is $0$. That is,


*

*$f^{-1}(0)=1$,

*$f^{-1}(1)=0$.


However, the preimages under $f$ of these as sets of ordinals are


*

*$f^{-1}[0]=f^{-1}[\varnothing]=\varnothing=0$,

*$f^{-1}[1]=f^{-1}[\{0\}]=\{f^{-1}(0)\}=\{1\}$.


But $0,1,\{1\}$ are all distinct sets, so $f^{-1}(0)\ne f^{-1}[0]$ and $f^{-1}(1)\ne f^{-1}[1]$. So these sorts of clashes can occur in situations where you have set-valued functions of sets and thus need to differentiate between applying inverse functions and preimages. It's worth mentioning that the ambiguity may arise in the forward direction with plain-old images just as well, so this is just a general feature of set-valued mappings of sets: $f(x)$ and $\{f(u):u\in x\}$ need not be the same set.
However elsewhere in mathematics it is generally unlikely one happens across such issues, so most don't need to bother with brackets as no real ambiguity will arise.
A: It is never helpful to express symbolically what can be simply stated.  The mathematical sentence $$\Phi^{-1}\left[\{\Phi(g)\}\right]=gH=Hg$$
is equivalent to the much easier sentence $$gH=Hg\text{ is the preimage of }\Phi(g)\text{ under }\Phi.$$
@amWhy's answer explains what all the various parenthesis mean.  I will elaborate on the group theoretic meaning.  In this context, $H$ is normal in $G$, and $\Phi$ is the homomorphism $$\Phi:G\rightarrow G/H\hspace{20pt}:\hspace{20pt}\Phi(g)=gH.$$
The point the text is trying to make is that any element $g^\prime$ in the coset $gH$ other than $g$ will also map to the coset $gH$ under $\Phi$.  In other words, $$gH=g^\prime H\Leftrightarrow \Phi(g)=\Phi(g^\prime).$$
This is a tautological statement, but intuitively this demonstrates that $\Phi$ represents the division of $G$ into a set of $H$-cosets (which are then imbued with a group law).  When the text refers to $gH$ being "collapsed into a point," they mean that the subset $gH$ of $G$ is only one element of $G/H$, whereas it is $|gH|$ elements in $G$.  Under the coset multiplication group law, it is sometimes possible to visualize this geometrically, for example the canonical homomorphism $\mathbb{R}^2\rightarrow \mathbb{R}^2/\left(\mathbb{R}\oplus 0\right)$ may be viewed as taking lines to points.
