Affine subspace $V\subseteq S(n)$ invariant under group action $x\mapsto gxg^{\mathrm{T}}$ of $G\subseteq GL(n, \mathbb{R})$ Given an affine subspace $V\subseteq S(n)$, where $S(n)$ is the real symmetric $n\times n$ matrices, is there a neat characterization of the maximal subgroup $G\subseteq GL(n, \mathbb{R})$ (with the group operation of left multiplication) such that $V$ is invariant (as a set) under the action $x\mapsto gxg^{\mathrm{T}}$ of $G$?
I took the very basic case of $n = 2$ and played around with some such subspaces and solved for the elements of $G$ by hand, and nothing really popped out at me, but I wonder if there's anything general we can say about $G$ if we have a parametrization of $V$, e.g. $$V = \left\{M_0+\sum_{i=1}^m t_iM_i : (t_1, \ldots, t_m)\in \mathbb{R}^m\right\}$$ for some $M_0, \ldots, M_m\in S(n)$.
Thanks for everyone's insights!
 A: The action is a similarity transformation and hence a rotation and scale. You are taking an affine subspace of S(n) and asking what the maximum similarity transformation group is.
Let $v \in V$ so that $v = s + t$ where $s,t \in S(n)$ then $g \in GL(n, \mathbb{R})$ is matrix multiplication: $w = gvg^T = gsg^T + gtg^T$ and you're asking what is the largest subgroup of $GL(n, \mathbb{R})$ that will give $w \in V$.
Clearly then $gtg^T = t$ since else we will end up in a different affine subspace. So $gt = tg$ or $[g,t] = 0$. Furthermore, once this is established $gsg^T$ must be in the translated kernel of $G$.
Essentially $V = g(V - t)g^T + t = gVg^T - gtg^T + t$. G must leave t and V fixed. There is only one fixed point in a similarity transform and that is the origin.
The problem is that $GL(n,\mathbb{R})$ will not be able to capture this since you need to be able to represent an affine transformation. This require V to be at least a proper subspace. From this, you just take the subgroup of $GL(n-1,\mathbb{R})$ and then add in the affine translation component. Invertability is not changed.
The problem is really no different the non-affine case but one has to go down in dimension. Most likely the treason you had issues with dimension 2 is end up with scalars as you need to $GL(n+1,\mathbb{R})$ to compensate for the affine component.
A very simple way to think about this is that affine vector spaces are simply non-affine vector spaces in one extra dimension(by adding a dimension the translation component of "affineness" becomes just a standard vector component and we can ignore the difference).
