The normalizer of a subset of a group is the part of the group that commutes with the subset (but not necessarily holding the element of the subset fixed): $g$ in the normalizer of $H \subset G$ means $gH = Hg$, or more specifically, $gh_1 = h_2 g$ for some $h_1, h_2 \in H$.
The centralizer of a subset of a group is the part of the group that commutes with the subset elementwise (i.e., holding the element of the subset fixed): $g$ in the centralizer of $H \subset G$ means $gh = hg$ for all $h \in H$. Clearly, commuting elementwise implies commuting with the subset, so the normalizer of a subset contains the centralizer of that subset.
The center of a group is the part of the group that commutes with everything in the group. Commuting with everything implies commuting with elements of some subset, so the centralizer of a subset contains the center of the group.
Putting these together: $Z(G) \subset C_G(H) \subset N_G(H)$.
Regarding "$\leq$"... The centralizer and normalizer are defined on subsets. It may turn out that a subset is also a subgroup, but this is not required. (It will essentially always be the case that the subsets are subgroups once you are past the material introducing these definitions.) This is why I have used "$\subset$" in the above.