Burnside's lemma for colourings on a triangle-square Suppose that we have $n$ colours available and we paint the edges and vertices of an equilateral triangle (resp. square) and we want to count how many colourings there are, up to equivalence, where two colourings are equivalent if one can be transformed into the other by applying a symmetry of the triangle (resp. square). How many colourings are there?
I know how to use Burnside's lemma to count the colourings when we only paint either the edges or the vertices of a regular $n$-gon, but not both. In the above problem, I initally thought that the case of painting the vertices and edges of a triangle was exactly the same as painting just the vertices of a regular hexagon, and similarly I considered working with a regular octagon in the place of the vertices and edges of a square. It seems intuitive, but something is not convincing me: Is the group of symmetries of the vertices and edges of a regular $n$-gon isomorphic to the group of symmetries of just the edges of a regular $2n$-gon (the dihedral group of $4n$ elements). Another possibility that passed through my mind was: just multiply the orbits of the action of $D_3$ by $2$ to get the colourings when we paint the vertices and edges.
Is there any merit in my thoughts? If so, please point out my mistakes. Thanks in advance, and sorry for the long text!
 A: For the triangle case, the symmetry group will still be $D_3$ (of six elements), and not of that a hexagon. The reason is you cannot bring the vertex to an edge. With that in mind, we consider $D_3$ acting on $X$, all possible colorings of vertices and edges with $n$ colors, so $|X| = n^6$ (before considering symmetries). Applying Burnside's theorem, we have the number of orbits $|X/D_3|$ to be
$$|X/D_3| = \frac{1}{|D_3|} \sum_{g\in D_3} |Fix(g)|,$$
where $Fix(g)$ is set of colorings fixed by $g$. Now we analyze this a bit, with $D_3 = \{1,r,r^2 , f_1,f_2,f_3\}$, where $r^i$ are the rotations and $f_j$ are the flips. Then
$|Fix(1)| = n^6$;
$|Fix(r)| = |Fix(r^2)| = n^2$ (one color choice for the vertices, and another color choice for the edges);
$|Fix(f_j)|=n^4$ (one color choice for the vertex where the line of symmetry goes through, one color choice for the edge where the line of symmetry goes across, one color choice for  the pair of vertices across the line of symmetry, and one color choice for the pair of edges across the line of symmetry).
Hence $|X/D_3| = \frac{1}{6}(n^6+ 2n^2 + 3n^4)=\frac{1}{6}n^2(n^4+2+3n^2)=\frac{1}{6}n^2(n^2+2)(n^2+1)$. Here I factor it to see that indeed we get an integer as $n^2(n^2+2)(n^2+1)$ is a multiple of 6.
For the square case you similarly consider the symmetry of a square $D_4$ with eight element, and not of an octagon for the same reason that we cannot bring a vertex onto an edge.
A: There are  two possibilities  here, rotational symmetry  (necklace) or
dihedral  symmetry (bracelet).  For the  first one  we have  the cycle
index of the cyclic group:
$$Z(C_n) = \frac{1}{n} \sum_{d|n} \varphi(d) a_d^{n/d}.$$
For second one we have the cycle index of the dihedral group
$$Z(D_n) =
\frac{1}{2} Z(C_n) +
\begin{cases}
\frac{1}{2} a_1 a_2^{(n-1)/2} & n \text{ odd} \\
\frac{1}{4} \left( a_1^2 a_2^{n/2-1} + a_2^{n/2} \right)
& n \text{ even.}
\end{cases}$$
With rotations the cycle structure induced by the vertices on the edges
is the same for vertices and edges  and with reflections the two types of
reflections are swapped when $n$  is even (use $a$  for vertices and $b$
for edges; we use the variable $n$ for the number  of vertices and $M$
for the number of colors). We thus obtain
$$Z(C'_n) = \frac{1}{n} \sum_{d|n} \varphi(d) a_d^{n/d} b_d^{n/d}.$$
and for the dihedral group
$$Z(D'_n) =
\frac{1}{2} Z(C'_n) +
\begin{cases}
\frac{1}{2} a_1 a_2^{(n-1)/2} b_1 b_2^{(n-1)/2}
& n \text{ odd} \\
\frac{1}{4}
\left( a_1^2 a_2^{n/2-1}  b_2^{n/2}
+ a_2^{n/2} b_1^2 b_2^{n/2-1} \right)
& n \text{ even.}
\end{cases}$$
Thus we have by Burnside for colorings with at most $M$ colors
(for all permutations of the group in question to fix a coloring it must be constant on each cycle so we have $M$ choices for the equal color of all slots on the cycle giving the substitutions $a_d = b_d = M$):
$$Z(C'_n; a_1 = M, b_1 = M, \ldots)
= \frac{1}{n} \sum_{d|n} \varphi(d) M^{2n/d}$$
and for the dihedral group
$$Z(D'_n; a_1 = M, b_1 = M, \ldots) =
\frac{1}{2} Z(C'_n; a_1 = M, b_2 = M, \ldots)
+ \frac{1}{2} M^{n+1}
\\ = \frac{1}{2n} \sum_{d|n} \varphi(d) M^{2n/d}
+ \frac{1}{2} M^{n+1}
.$$
This gives with $M=2$ (two colors) the sequence
$$4, 9, 20, 51, 136, 414, 1300, 4371, 15084, 53508, \ldots$$
which points us to OEIS A161221, where these
data are confirmed. We also obtain with $n=3$ (triangle)
$$\frac{1}{6} (M^6 + 2 M^2) + \frac{1}{2} M^4.$$
This factors as
$$\frac{1}{6} M^2 (M^2+1) (M^2+2)$$
as pointed out in the accepted answer. We are asked to do squares
as well and obtain
$$\frac{1}{8} (M^8 + M^4 + 2 M^2) + \frac{1}{2} M^5
= \frac{1}{8} (M^8 + 4 M^5 + M^4 + 2 M^2).$$

N.B. the two types of  reflections when $n$ is even are first, those that
fix two opposite  vertices and second, those that fix two opposite edges.

There is more on this at
MSE 3216181.
