If S is a normal subgroup, identify the quotient group G/S. What are the $\varphi(G)$'s? The following is an exercise from Artin's Algebra: (Kiefer Sutherland's voice)

Let G be the group of upper triangular real matrices $\begin{bmatrix}
a & b\\ 
0 & d
\end{bmatrix}$ with a and d different from zero. For each of the following subsets, determine whether or not S is a subgroup, and whether or not S is a normal subgroup. If S is a normal subgroup, identify the quotient group $G/S$.
(i) S is the subset defined by b = O. 
(ii) S is the subset defined by d = 1. 
(iii) S is the subset defined by a = d. 

The following questions are related:
How the quotient group of following matrix group looks like?
Proving that $G/N$ is an abelian group
I found a solution by Blake Griffith linked here.
The following is a screenshot of the solution:


The following are my questions:


*

*How is $S$ is the $\ker$ for iii? I computed the kernel to be $$\begin{bmatrix}
x_{11} & x_{12}\\ 
0 & x_{22}
\end{bmatrix}$$
where $x_{21} = 0, x_{12} \in \mathbb R, x_{11} \ne 0 \ne x_{22}$. But how do we conclude $x_{11} = x_{22}$? I think I'm missing something slightly because the solution was (ii) was incredibly inventing:


*Is this correction to ii correct?



For ii, we set
  $$\varphi(X)=X\pmatrix{\frac{1}{x_{11}}&\frac{x_{12}}{x_{11}}\\0&1}=\pmatrix{1&2x_{12}\\
 \frac{x_{21}}{x_{11}}&\frac{x_{21}x_{12}}{x_{11}}+x_{22}}=\pmatrix{1&0\\0&1}$$
  to get $S$.

It was pointed out this is wrong, but I think I can correct this.

For ii, we set
  $$\varphi(X)=X\pmatrix{\frac{1}{x_{11}}&\frac{-x_{12}}{x_{11}}\\0&1}=\pmatrix{1&0\\
 \frac{x_{21}}{x_{11}}&\frac{-x_{21}x_{12}}{x_{11}}+x_{22}}=\pmatrix{1&0\\0&1}$$
  to get $S$.



*Does the $G'$ matter? I think it doesn't, but I want to confirm.

*What are the $\varphi(G)$'s?
For ii, I think $\varphi(G)$ is all of $d \ne 0, b \in \mathbb R$ in $$\begin{bmatrix}
a & b\\ 
0 & d
\end{bmatrix}\begin{bmatrix}
\frac 1 a & \frac b a \\ 
0 & 1
\end{bmatrix}=\begin{bmatrix}
1 & 2b\\ 
0 & d
\end{bmatrix}$$
For iii, I think $\varphi(G)=I_2$ because
$$\begin{bmatrix}
a & b\\ 
0 & d
\end{bmatrix}\begin{bmatrix}
\frac 1 a & \frac b {-ad} \\ 
0 & \frac{1}{d}
\end{bmatrix}=\begin{bmatrix}
1 & 0\\ 
0 & 1
\end{bmatrix}$$
What does it mean that $\varphi(G)$ is just the identity of $G'$? $|G/S|=1$ I think
 A: Though you didn't question this, I found it rather strange that the verifications of being closed under multiplication are actually reduced to verification of being closed under squaring an element. I think it's not enough in general. However, showing that these sets are closed under multiplication takes the same effort..
To your questions,


*

*Neither of these maps $\phi$ seem to work. I guess, they are not even group homomorphisms.
As it stands - you correctly calculated - $\phi\pmatrix{a&b\\0&d}=\pmatrix{1&2b\\0&d}$ for ii) and $\phi\pmatrix{a&b\\0&d}=\pmatrix{1&0\\0&1}$ for iii). 
Neither does produce the corresponding set $S$ as the preimage of the identity matrix.
I thought, maybe that $X$ is superfluous in the definitions, so speaking about e.g. $\phi\pmatrix{a&b\\0&d}=\pmatrix{1/a&b/a\\0&1}$, but those don't work either.

*$G'$ doesn't really matter, can be any ambient group. It's the image of $\phi$ what matters. (Once we find better candidates for the $\phi$'s).

*Your calculations and conclusions are correct. However, we have to find better $\phi$'s, or just describe the quotients as they are.
A: There are a few problems in the proposed solution.
The remark that
$$
\begin{bmatrix} a & b \\ 0 & d \end{bmatrix}^{-1}=
\begin{bmatrix} a^{-1} & -ba^{-1}d^{-1} \\ 0 & d^{-1} \end{bmatrix}
$$
is correct and useful.
Case (i), call the set $S_1$
The set $S_1$ is not empty, because $I$ (the identity matrix) belongs to it. Also the inverse of every element in $S_1$ is again in $S_1$.
Closure under multiplication requires two arbitrary elements of $S_1$:
$$
\begin{bmatrix} a & 0 \\ 0 & d \end{bmatrix}^{-1}
\begin{bmatrix} a' & 0 \\ 0 & d' \end{bmatrix}^{-1}=
\begin{bmatrix} aa' & 0 \\ 0 & dd' \end{bmatrix}^{-1}\in S_1
$$
In order to show $S_1$ is not normal, an explicit example should be given. The computation is correct, but stating that
$$
gsg^{-1}=\begin{bmatrix} a & b'(d-a)d' \\ 0 & d \end{bmatrix}
$$
is not sufficient; however we can take $a=2$, $d=1$, $a'=b'=d'=1$ and we'd be done.
Note that choosing $a\ne d$ is very important: the same computation applied to the set in (iii) shows that subgroup is normal!
Case (ii)
This case, apart from the glitch about the product, is correct.
Case (iii)
The same as above applies.
Identifying the quotient groups
For $S_2$, the group in (ii), you can observe that
$$
\begin{bmatrix} a & b \\ 0 & d \end{bmatrix}=
\begin{bmatrix} 1 & 1 \\ 0 & d \end{bmatrix}
\begin{bmatrix} a & b-1 \\ 0 & 1 \end{bmatrix}
$$
and that the first term in the right-hand side is in $S_2$, depending only on $d$. This gives the idea to consider the map $\varphi_2\colon S\to\mathbb{R}^*$ (nonzero real numbers with respect to multiplication) defined by
$$
\varphi_2\left(\begin{bmatrix} a & b \\ 0 & d \end{bmatrix}\right)=d
$$
This is seen to be a (surjective) homomorphism, with kernel exactly $S_2$.
