In group theory, are the operations left to right or right to left? I got points taken off on homework because for the dihedral group $D_3$, I needed to find the left coset of a subgroup $H$. So for "flip $\cdot$ rotation" I did the flip first and then the rotation, but apparently it's the other way around. Does this hold in general? Reading right to left when we manipulate an object?
 A: This is the difference between a right and a left action.
When we say that the elements of the dihedral group $D_3$ correspond to "flipping" or "rotating", we are implicitly thinking about some object (here, a triangle $T$) which is being flipped or rotated.
Now we say that $D_3$ acts on $T$ from the left (and we call this a left action) when we write
$\mathsf{flip} \cdot T$ to mean that we apply "flip" to the triangle $T$.
Dually, we say that $D_3$ acts on $T$ from the right (and we call this a right action) when we write $T \cdot \mathsf{flip}$ to mean we apply "flip" to $T$.
Obviously it doesn't matter which choice we make, but once we make a choice we have to stick with it! People tend to use left actions (and indeed, this is what your professor was expecting. Particularly since you're using left cosets) because we're used to writing $f(x)$ for function application. So it feels more "normal" to write $\mathsf{flip} \cdot T$ for "flip applied to $T$".
The price we pay, though (and the reason many authors prefer right actions), is that composition feels "backwards".
Intuitively, we would expect (as you did) that $\mathsf{flip} \cdot \mathsf{rotate}$ should do a flip first, then a rotation. But let's see what this group element does to $T$:
$$(\mathsf{flip} \cdot \mathsf{rotate}) \cdot T = \mathsf{flip} \cdot (\mathsf{rotate} \cdot T)$$
So, counterintuitively, $\mathsf{flip} \cdot \mathsf{rotate}$ applied to $T$ is the same thing as first rotating $T$, then flipping the result. We do the thing on the right first! This is exactly like function composition, where $(f \circ g)(x) = f(g(x))$ does $g$ first then $f$.
If instead, we used a right action, then we see:
$$
T \cdot (\mathsf{flip} \cdot \mathsf{rotate}) = (T \cdot \mathsf{flip}) \cdot \mathsf{rotate}
$$
Now the price we pay is slightly more esoteric notation (it looks odd to write $(x)f$ for "$f$ applied to $x$") but the benefit is that now composition goes the way we expect. Notice that in a right action, $\mathsf{flip} \cdot \mathsf{rotate}$ applied to $T$ is the same thing as flipping first, then rotating!
There was actually a push in the mid-to-late 20th century (particularly by the category theorists) to start writing function application on the right in order to make composition more intuitive. It's tricky though, because even though many people think it's the "right" thing to do (pun very much intended), it's very hard to break from years of tradition and muscle memory. I'm not surprised that it didn't really catch on.

I hope this helps ^_^
