What is an example of a situation where AB is not subgroup of G, when A, B are subgroups of G? What is an example of a situation where AB is not subgroup of G, when A, B are subgroups of G?
My first instinct is always to go for some dihedral group or other...But I could not find an example of a case when the above is true. 
Maybe I am doing it wrong...
 A: A generic type of example is when $G$ is a finite group which more than one Sylow $p$-subgroup for som prime $p.$ Suppose then that $A$ and $B$ are different Sylow $p$-subgroups of a finite group $G$, say $|A| = |B| = p^{a}$, so that $p^{a+1}$ does not divide $|G|.$ Then the SET
$AB = \{ab: a \in A, b \in B \}$ has cardinality $\frac{|A| |B|}{|A \cap B|}$ which is a power of $p$ and is at least $p^{a+1}$ since $A \cap B \neq A.$ Hence $AB$ can't be a subgroup of $G$, as the order of a subgroup of $G$ divides $|G|$ by Lagrange's theorem, whereas $p^{a+1}$ 
(or any higher power of $p$) does not divide $|G|$.
   The example Don Antonio gives is an instance of this. Another example is given by $G = A_{5}$, $A = \langle (12345) \rangle$, $B = \langle (13245) \rangle$, where $AB$ has order $25$, so $AB$ is not a subgroup of $A_{5}.$
A: For example
$$A=\{(1),(12)\}\,\,,\,\,B=\{(1),(13)\}\leq S_3\Longrightarrow AB=\{(1),(12),(13),(132)\}\rlap{\;\,/}\leq S_3$$
A: This is an in page 40 from Hungerford's book Algebra, however; touchable examples above are more interesting to me.

Let $G$ be a group of order $p^{k}m,$ with $p$ prime and $(p,m)=1.$ Let $H$ be a subgroup of order $p^{k}$ and $K$ a subgroup of order $p^{d}$, with $0<d\leq k$ and $K\nsubseteq H.$ Show that $HK$ is not a subgroup of $G$.

You can solve it by using this fact that: $$\displaystyle |HK|=\frac{|H||K|}{|H\cap K|}$$
