Where is my mistake to count $\#Hom(D_4, S_3)$? Please help me. I am willing to find the number of group homomorphisms from $D_4$ to $S_3$. 
Now we know that $$D_4=\langle r_4, f_4: r_4^4=f_4^2=(r_4f_4)^2=e \rangle.$$ 
So if $\phi:D_4\rightarrow S_3$ be any group homomorphism then we need to find the correct possibilities of $\phi(r_4)$ and $\phi(f_4)$ satisfying the criteria $$\phi((r_4f_4)^2)=\phi(e)=\varepsilon..................(\ast)$$ 
From here my dilemma starts. Shall we reduce ($\ast$) in the following way: 
(1) $\varepsilon=\phi((r_4f_4)^2)=\phi((r_4f_4)(r_4f_4))=(\phi(r_4f_4))^2=(\phi(r_4)\phi(f_4))^2$
or shall we do like this:
(2) $\varepsilon=\phi((r_4f_4)^2)=\phi((r_4f_4)(r_4f_4))=(\phi(r_4f_4))^2\Rightarrow |\phi(r_4f_4)|$ divides 2.
Trouble is: (1) gives completely different answer from (2). 
Let me show you my work. May be you can tell me if I am making any mistake. 
Lets choose (1). 
Here $\phi(f_4), \phi(r_4)$ have the choices $\{\varepsilon, (12), (13), (23)\}$. 
If $\phi(f_4)=\varepsilon$ then $\phi(r_4)\in\{\varepsilon, (12), (13), (23)\}$
satisfies $(\ast)$ and we get 4 possibilities.
If $\phi(f_4)=(12)$ then $\phi(r_4)$ can be any of $\{\varepsilon, (12), (23), (13)\}$ to satisfy $(\ast)$. Total 4 choice again. Similarly $\phi(f_4)=(13), (23)$ will also give us 4=4=8 choices. 
Total 4+4+4+4=16 possible group homomorphisms here we get if we consider the (1) as reduced form of $(\ast)$. 
What if we consider (2) ?In this case when $\phi(f_4)=\varepsilon$ as before we get $\phi(r_4)\in \{\varepsilon, (12), (13), (23)\}$ --- 4 choices. 
But if $\phi(f_4)=(12)$ then $\phi(r_4)$ will be only $\varepsilon, (12)$ to satisfy $(\ast)$ in (2). cause if $\phi(r_4)=(13)$ then $|\phi(r_4f_4)|=|\phi(r_4)\phi(f_4)|=|(12)(13)|=|(132)|=3$ divides 2, contradiction. Similarly $\phi(r_4)\neq (23)$. So only 2 cases if $\phi(f_4)=(12)$. Similarly $\phi(f_4)=(13), (23)$ will bring 2+2=4 more cases. 
Total 4+2+2+2=10 choices. 
Now where am I making mistake ? Please tell me
 A: First of all, the presentation you have written is for $D_4$, not $D_3$.  
(2) is correct (assuming you change $D_3$ to $D_4$ everywhere).  A homomorphism has to satisfy all the relations in the original presentation.  Since the original group had $$r_4^4 = f_4^2 = (r_4 f_4)^2 = e$$ then the images of $r_4$ and $f_4$ under a homomorphism must satisfy $$\phi(r_4)^4 = \phi(f_4)^2 = \phi(r_4 f_4)^2 = e.$$
A: Up to the implication '$\implies |\phi(r_4f_4)^2|$ divides $2$' (which is in fact an equivalence) both (1) and (2) are identical and will give you the same result.
Let us have a look at (1). If $\phi(f_4) = (1,2)$, then $\phi(r_4) = (2,3)$ is no option, for you would get $e = ((1,2)(2,3))^2 = (1,3,2)$ which is not the case. Similarly, $(1,3)$ is no option either and also for the other two nontrivial options for $\phi(f_4)$ you will have to discard two of the possibilities you wrote down which will give you $10$ possibilities as in (2).
I am guessing here, but you might have confused $(\phi(r_4)\phi(f_4))^2$ with $\phi(r_4)^2\phi(f_4)^2$. Note that $S_3$ is not abelian so that these two expressions are not identical in general.
