Give an example of a group $G$ and two of its elements $x$ and $y$ such that $|x|=|y|=2$ and $|xy|=3$. I tried the group of integers under the addition mod $n$ but realized that group can only have one elements whose order is $2$. For addition modulo $6$, that element would be $3$. For addition modulo $12$, that element would be $6$ and so on.
I also tried my luck finding such group from integers under multiplication modulo $n$ but still couldn't find any. 
But I am certain that the group has to be finite. Then do I approach this problem by using properties of finite groups to find such group?  
 A: The group doesn’t have to be finite, but it might as well be.
The key thing to realize is that if elements $g$ and $h$ commute, then the order of $gh$ divides the least common multiple of the orders of $g$ and $h$. Hence, if $g \neq h$ are of order $2$ in an abelian group, then $gh$ also has order $2$ (this also applies to elements that just-so-happen to commute, even if the entire group isn’t abelian). So, it won’t be productive to look at abelian groups.
Aside from that, there’s no special group theoretic thing to use, unless you happen to be familiar with particular families of groups.
In fact (as another answer points out) the desired phenomenon occurs in the smallest nonabelian group, so you don’t have to look far, once you've decided that "nonabelianness" is an essential property.
A: Hint take the symmetric group $S_3$ and the permutations $(12)$ and  $(23)$.
A: Firstly, note that in general the orders of $x$, $y$ and $xy$ do not uniquely define the group (up to isomorphism). For example, both the Quaternion group and the direct product of two cyclic groups of order $4$, $\mathbb{Z}/4\mathbb{Z}\times \mathbb{Z}/4\mathbb{Z}$, are generated by elements of order $4$ whole product also has order $4$ (in fact, one copy of $\mathbb{Z}/4\mathbb{Z}$ also works, but that feels like cheating!). So someone has said that $S_3$ works...but you should wonder: are there other examples?!
No. There are no other examples, and the group you describe is uniquely defined (up to isomorphism) as $S_3$. Another answer points out that $S_3$ works. To see that this is the only group which works, firstly note that any group with elements of order both $2$ and $3$ must have order dividing $6$. Then I prove below that there are at most $6$ elements in your group, and so the order of your group is precisely $6$. There are two groups of order $6$ (one abelian and cyclic, and one non-abelian), and your "rules" hold for one but not the other. Hence, the group you describe must be $S_3$.
To see that there are at most $6$ elements, note that there are clearly at most $11$ elements: $$\{1, x, xy, xyx, xyxy, xyxyx\}\cup \{1, y, yx, yxy, yxyx, yxyxy\}$$
(for example, we stopped at $xyxyx$ as $xyxyxy=1$). However, we have double-counted:
$$
\begin{align*}
xyx=(xyx)^{-1}&=yxy,\\ xyxy=(yxyx)^{-1}&=yx,\\ yxyx=(xyxy)^{-1}&=xy,\\ xyxyx=(xyxyx)^{-1}&=y\\
yxyxy=(yxyxy)^{-1}&=x
\end{align*}$$
So we have double-counted $5$ elements, which gives $11-5=6$ elements, as claimed:
$$
\{1, x, y, xy, yx, xyx\}
$$
