Permutation Group / Orbit counting principle A stained glass window consists of nine squares of glass in a 3x3 array. Of the nine squares, k are red, the rest blue. A set of windows is produced such that any possible window can be formed in just one way by rotating and/or turning over one of the windows in the set. Altogether there are more than 100 red squares in the set. Find k.
first, there are 8 Isometries of a square.
Identity, three rotations (90,-90,180)
four reflections (vertical, horizontal, two diagonal axis).
let G be the permutation group, then |G|=8, and I can find fix(g) for every g.
can someone give me a hint of how to proceed from there.
 A: For $k=0$ or $k=9$, there is only one orbit, and for $k=1$ or $k=8$ there are only $3$ orbits, so these are right out. For $k=2$ or $k=7$, a direct count yields $8$ orbits ($5$ including a minority corner and $3$ not), and $8\cdot7=56\le100$, so these are also excluded. That leaves $k=3$ to $6$.
For $k=3$ or $k=6$, the identity leaves $\binom93=84$ windows invariant, a rotation through $\frac\pi2$ leaves no windows invariant, a rotation through $\pi$ leaves $4$ windows invariant (the centre and one of four opposite pairs must be in the minority), and a reflection leaves $1+3\cdot3=10$ windows invariant (either all three squares on the axis or one of them and one of the three reflected pairs must be in the minority). Thus by Burnside's lemma there are $\frac18(84+2\cdot0+4+4\cdot10)=\frac{128}8=16$ orbits, not quite enough for $100$ red squares for $k=6$.
For $k=4$ or $k=5$, the identity leaves $\binom94=126$ windows invariant, a rotation through $\frac\pi2$ leaves $2$ windows invariant, a rotation through $\pi$ leaves $\binom42=6$ windows invariant (any two of the four opposite pairs must be in the minority), and a reflection leaves $\binom32\binom31+\binom32=12$ windows invariant (either two squares on the axis and one reflected pair must be in the minority, or two reflected pairs). Thus there are $\frac18(126+2\cdot2+6+4\cdot12)=\frac{184}8=23$ orbits, which is not quite enough with $k=4$ but yields $5\cdot23=115$ red squares for $k=5$.
To summarize, there are $0,3,16,48,92,115,96,56,24,9$ red squares for $k=0$ to $9$ respectively.
A: I upvoted the first answer but I would like to show how to compute the
cycle index  $Z(G)$ of the group  $G$ of symmetries of  the square and
apply the Polya Enumeration Theorem to this problem.
We  need  to  enumerate   and  factor into cycles  the  eight permutations  that contribute to $Z(G).$
There is the identity, which contributes
$$a_1^9.$$
The two $90$ degree rotations contribute
$$2a_1 a_4^2.$$
The $180$ degree rotation contributes
$$a_1 a_2^4.$$
The vertical and horizontal reflections contribute
$$2a_1^3 a_2^3.$$
The reflections in a diagonal contribute
$$2a_1^3 a_2^3.$$
This yields the cycle index
$$Z(G) = \frac{1}{8}
(a_1^9 + 2a_1 a_4^2 + a_1 a_2^4 + 4a_1^3 a_2^3).$$
As we are interested in the red squares we evaluate
$$Z(G)(1+R)$$
to get
$$1/8\, \left( 1+R \right) ^{9}+1/2\, \left( 1+R \right) ^{3}
 \left( {R}^{2}+1 \right) ^{3}+1/8\, \left( 1+R \right)
 \left( {R}^{2}+1 \right) ^{4}\\+1/4\, \left( 1+R \right)
 \left( {R}^{4}+1 \right) ^{2}$$
which is
$${R}^{9}+3\,{R}^{8}+8\,{R}^{7}+16\,{R}^{6}+23\,{R}^{5}+23\,{R}^
{4}+16\,{R}^{3}+8\,{R}^{2}+3\,R+1.$$
This is  the classification of the  orbits according to  the number of
red squares.  Differentiate and  multiply by $R$  to obtain  the total
count of the squares, which yields
$$9\,{R}^{9}+24\,{R}^{8}+56\,{R}^{7}+96\,{R}^{6}+115\,{R}^{5}+92
\,{R}^{4}+48\,{R}^{3}+16\,{R}^{2}+3\,R$$
which matches the accepted answer.
