If you are not allowing "degenerate" triangles and you do not want to label the vertices, then the space of triangles is the same as the space of unordered triples of distinct points on a circle. The space of ordered triples of distinct points is a subset of $S^1\times S^1\times S^1$ (namely $(S^1)^3$ minus the "diagonal"), and the set of unordered triples is the quotient of this space by the action of the symmetric group $S_3$ that permutes the coordinates (that is, it equates $(x,y,z)$ with $(x,z,y)$ and $(y,x,z)$ and so on).
This is commonly called the "configuration space" of 3 points on a circle.
However, this is just the space of triangles. It is not a description of a group of transformations of the space. I am not sure that the transformations you are describing form a group, unless you label the points and allow degenerate triangles.