What are useful tricks for determining whether groups are isomorphic? In general, it is not too hard to find isomorphisms between two groups when their order is relatively low. However, as their orders grow, it becomes increasingly irritating to write down their entire Cayley tables and such. Is there a set of tricks that is generally useful when trying to prove that two groups are actually isomorphic? After all, it usually seems easier to prove that they aren't, as you just need to point out one property that doesn't correspond...
Example: in Armstrong's Groups and Symmetry, it is asked to show that the dihedral group of order 8 and the subgroup of S4 generated by (1234) and (24) are isomorphic. It is easy to send D4's "single-rotation" element r to S4's (1234) and D4's "flipping" element s to S4's (24), as they are all part of the generating set and their orders coincide, but what is the way to go from here? Also, how far should one go in showing the isomorphism - might pointing out the correspondence in generating elements even be enough?
 A: There are a few ways to show that a homomorphism of groups is an isomorphism:


*

*A homomorphism which is a bijection of sets is an isomorphism.

*A homomorphism with a two-sided inverse is an isomorphism.

*A homomorphism which is surjective as a map of sets and with a trivial kernel is an isomorphism.


Of course, sometimes the hard part is showing that you have a homomorphism at all. For your example of $D_4$ and $S_4$, you need to show the map you specify extends to a homomorphism. It's automatically surjective because you've hit every element in the generating set. Then it suffices to show that it has a trivial kernel. However, the easiest way to show that $D_4$ embeds as a subgroup of $S_4$ is to show that it acts faithfully on a set of 4 elements. This is easy: $D_4$ naturally acts on the vertices of a square, and it's certainly faithful because there's no non-trivial element of $D_8$ which acts trivially.
A: The usual way to show isomorphy between groups is indeed to simply construct the isomorphism, i.e. construct a function $\varphi$ that is a bijection and morphism. 
However sometimes you can resort to neater 'tricks', by considering actions of groups and constructing equivalent permutation representations. For instance to show that $Aut(\mathbf C_2\times \mathbf C_2) \cong \mathbf S_3 \cong \mathbf D_6$ one could notice that all three groups act on three elements. The first one actually consists of all permutations of the non-trivial elements of  $C_2\times \mathbf C_2$, (1,0), (0,1) and (1,1), the second group is by definition the group of all permutations of $\{1,2,3\}$ and the third group acts on a triangle in the plane where once again it consists of all possible permutations of the three vertices of the triangle. 
All three groups act faithful (only the identity acts trivially) therefore any bijection between these sets (the non-trivial elements of $\mathbf C_2\times \mathbf C_2$, the set $\{1,2,3\}$, the angles of a triangle) will naturally induce an isomorphism between these groups. It's not really an alternative to constructing the isomorphism ofcourse, it just tells you where to start looking for one.
Another strategy that I've seen at work when dealing with groups defined by presentations is to first construct an epimorphism (which is rather easy, one only has to verify that all relations still hold) and then show by considerations about the group (e.g. simplicity) that its kernel must be trivial.
However, it's often very hard to think of such things. Here's something I read: During the golden years of discovery of sporadic simple groups two groups of equal order were constructed and even though this was suspected and quite some non-trivial things about these groups had been proven, it remained unsure for quite another while whether or not they were actually equal. (It turned out they were. It would be interesting if someone had a reference of this fact, I forgot where I read it.)
--- added:
Here's how this may help to solve your particular problem. Let $\mathbf D_8$ act on a square with vertices numbered $1,2,3,4$. Note that indeed $a = (1\ 2\ 3\ 4)$ and $b = (2\ 4)$ correspond to valid isometries of the square, let's call them $R$ (for rotation) and $D$ (for reflection along the diagonal). This implies that the map 
$$ \varphi \colon \langle R,D\rangle \to \langle a,b\rangle $$
is actually an isomorphism: each element of either groups uniquely corresponds to some permutation of $\{1,2,3,4\}$ resp. the vertices of the square. Therefore it is sufficient to show that $\mathbf D_8$ is generated by $R$ and $D$ (which is well-known) and you're done.
A: Proving that two groups are isomorphic is a provably hard problem, in the sense that the group isomorphism problem is undecidable. Thus there is literally no general algorithm for proving that two groups are isomorphic. To prove that two finite groups are isomorphic one can of course run through all possible maps between the two, but that's not fun in general.
For your particular example, there's not much to say. It depends on what you know about $D_4$. If you know a presentation of it, then you can prove that what you've defined is actually a homomorphism which is moreover surjective. Since the two groups have the same order, you're done.
For recognizing a finite group via a presentation you can sometimes use the Todd-Coxeter algorithm. 
