Consider the dihedral group $H^5_2$ acting as a group of symmetries of a regular pentagon. In particular, this group acts by permutations of the five vertices of the pentagon. Identify $H^5_2$ as a subgroup in the symmetric group $S_5$. Which permutations of 5 elements does it contain? Is this subgroup normal in $S_3$?

Do I need to show that $H^5_2$ acts on $S_5$? How can I better understand the concept of subgroups, action and normality of groups because I am still confused with the notations in the book. Please help.

$S_5$ consists of all permutations on a set of $5$ elements, or 'labels'. The easiest labels to work with here are probably the numbers 1,2,3,4,5. Now take your pentagon and label the vertices with these in some order. Now see what happens to the labels as you apply the action of the dihedral group.

You should see that the dihedral group will permute the labels around (the label at the top may change from 1 to 3, etc.), but not nearly as arbitrarily as $S_5$ can permute them. In this way you can see that the dihedral group naturally embeds as a subgroup of $S_5$, and you can also work out exactly which elements are in it by writing out the permutation of the labels the dihedral action produces.

Note that exactly which subgroup you embed $H_2^5$ as will depend on exactly how you label the vertices on the first labelled pentagon you start with.

If the dihedral group of 10 elements feels a bit large to work with here, try working with the dihedral group of order 8 and $S_4$ (note that $H_2^3$ and $S_3$ are isomorphic, so you can work with those as well but it may not be as illuminating of the difference between the two groups in general).

An alternative way, but which lacks the fairly nice geometric picture, is to simply look for elements in $S_5$ which have the same relations that the generators for $H_2^5$ do. If you just write out permutations $\sigma,\tau$ such that $\sigma^5=\tau^2=\operatorname{id}$ and $\tau\sigma\tau =\sigma^{-1}$, then automatically the subgroup $\langle \sigma,\tau\rangle$ is isomorphic to $H_2^5$.

As an aside, normally I would see the dihedral group of order 10 written as either $D_{10}$ or $D_5$, depending on the author.

Answering the rest of your question is covering a bit too much ground. I suggest going to your professor's office hours, or otherwise arranging time to talk with him. A teaching assistant, if there is one for the course, should also suffice. There are doubtlessly numerous webpages and other online resources dedicated to teaching the fundamentals of group theory which you could find through google. You could still use the stackexchange, but questions usually need to be a bit more concise and specific so we can produce good, reasonable-length answers.