Is the symmetry group of a compact subset of $\mathbb{R}^n$ closed? Let $K$ be a compact subset of $\mathbb{R}^n$. Let $G(K) = \{f \in \mathrm{Isom}~\mathbb{R}^n | f(K) = K\}$, $E=$ be the identity connected component of $G(K)$

*

*Is it true that $G(K)$ is a closed subgroup $\mathrm{Isom} ~\mathbb{R}^n$?

*Is it true that if $G$ is a closed subgroup $\mathrm{Isom} ~\mathbb{R}^n$, then $G/E$ is a discrete group?

*Is it true that for the exponentiation operation $\mathbb{Z} \times E \to E$ there is a natural continuation to the continuous operation $\mathbb{R} \times E \to E$ ? It seems to me that this implies that $E$ is isomorphic (as topological group) $SO(k)$, am I right?

*Please advise the best standard textbooks on topological groups / actions of topological groups on topological spaces. My knowledge of this is limited to the corresponding chapter in the book Elementary Topology. As far as I can tell Topological Groups and Related Structures not very relevant. I thought Topological Groups (Pontryagin) was not very good (maybe outdated), but I’ll be glad to hear that I jumped to conclusions.

 A: In general:

Lemma 1. Let $X$ be a topological space and $C\subseteq X$ a connected component. Then $C$ is closed.

Proof. Connected Components are Closed $\Box$

Lemma 2. Let $G$ be a topological group and $E\subseteq G$ be the identity component of $G$. Then $E$ is a closed normal subgroup of $G$.

Proof. Consider multiplication $M:G\times G\to G$, $M(g,h)=gh$ and the inverse map $I:G\to G$, $I(g)=g^{-1}$.

*

*By lemma 1 $E$ is closed.

*Since $E$ is connected then so is $E\times E$. Since $M$ is continuous then $M(E\times E)$ is connected. Since $M(e,e)=e$, $M(E\times E)$ is connected and $E$ is a connected component then $M(E\times E)\subseteq E$. In particular $E$ is closed under multiplication. Analogously $I(E)\subseteq E$. Thus $E$ is a subgroup.

*For any $h\in G$ consider the inner automorphism $f_h:G\to G$, $f_h(g)=hgh^{-1}$. Obviously $f_h$ is continuous and since $f_h(e)=e$ then analogously to point $2$ we get $f_h(E)\subseteq E$. Since this holds for any $h$ then $E$ is normal. $\Box$

Lemma 3. Let $G$ be a topological group and $E$ be the identity component. Then $G/E$ is totally disconnected.

Proof. This follows from the fact that the quotient map $\pi:G\to G/E$ is open and the fact that connected components of $G$ are precisely cosets over $E$. I leave details as an exercise. $\Box$


*

*Is it true that $G(K)$ is a closed subgroup $\mathrm{Isom} ~\mathbb{R}^n$?


Yes. Assume that $f\in\overline{G(K)}$ in $\mathrm{Isom} ~\mathbb{R}^n$. So there is a net $(f_\alpha) \in G(K)$ convergent to $f$. In particular for each $x\in K$ we have that $f_\alpha(x)\in K$ is a convergent net. Thus $f(x)=\lim f_\alpha(x)\in K$, and so $f\in G(K)$. $\Box$



*Is it true that if $G$ is a closed subgroup $\mathrm{Isom}~\mathbb{R}^n$, then $G/E$ is a discrete group?


By lemma 3 $G/E$ is always totally disconnected. Generally $G/H$ is discrete if and only if $H$ is an open subgroup of $G$.
Now in this particular case, our $E$ has to be contained in $G$ for $G/E$ to even make sense. Or maybe you meant $G/(G\cap E)$? Either way note that $\mathrm{Isom} ~\mathbb{R}^n$ is not connected, but its identity component is so called special Euclidean group $E^+(n)$ which is of index $2$. It can be shown that a closed subgroup of finite index has to be open. And thus $\mathrm{Isom} ~\mathbb{R}^n/E^+(n)$ is discrete (with two elements). And also every quotient $G/(G\cap E)$ has to be discrete, because $G\cap E$ is always open in $G$.
Another way to look at this is from Lie perspective (since $\mathrm{Isom} ~\mathbb{R}^n$ is a Lie group). A Lie group is a manifold, and connected components of manifolds (or any locally connected space) are open. Also since a closed subgroup of a Lie group is Lie as well, then this generalizes to any closed subgroup $G$ of $\mathrm{Isom} ~\mathbb{R}^n$ and its own identity component.



*Is it true that for the exponentiation operation $\mathbb{Z} \times E \to E$ there is a natural continuation to the continuous operation $\mathbb{R} \times E \to E$ ?


I assume that by exponentiation you mean $(n,f)\mapsto f^n$ map. Assume it is a group action, i.e. $(f^n)^m=f^{n+m}$. Then $x^2=x^{1+1}=(x^1)^1=x$. In particular $x$ is the identity element. And thus our function is a group action only when $E$ is trivial.
So the question is whether this exponentiation map can be extended to full $\mathbb{R}$? Obviously by Tietze any continuous function $\mathbb{Z}\times E\to E$ can be extended to $\mathbb{R}\times E\to E$. But if we want the additional "being a group action" property, then $E$ has to be trivial, and thus $\mathbb{R}$ can trivially act on it. That's the only case where it works.

It seems to me that this implies that $E$ is isomorphic (as topological group) $SO(k)$, am I right?

This depends on $K$. Consider $K=[0,1]\cup\{2\}\subseteq \mathbb{R}$. Then the only isometry $\mathbb{R}\to\mathbb{R}$ that maps $K$ to $K$ is the identity. Meaning both $G(K)$ and its identity component are trivial groups.
On the other hand the $\mathrm{Isom} ~\mathbb{R}^n$ group (without restrictions) has the special Euclidean group as the identity component. This group is isomorphic to a semidirect product of $SO(n)$ by $T(n)$ (the subgroup of all translations). So again not $SO(n)$. I'm not sure if there is $K$ such that the identity component of $G(K)$ is $SO(n)$ to be honest.
