When we're teaching students about categories, we usually give lots of examples of things that are categories but very few (or no!) examples of things that are not categories.
Part of this problem is that it's hard to find classes of maps that don't satisfy composition. Continuous maps compose to continuous maps, and likewise for open, closed, proper, or quotient maps; similarly for homeomorphisms and homotopy equivalences. We can try a geometrical setting, but again the composition of $C^k$ maps for $1 \le k \le \infty$ (or even $k = \omega$, the analytic maps) is again a $C^k$ map.
One class of maps that does work is the inclusions of normal subgroups: if $H \trianglelefteq K$ and $K \trianglelefteq G$, it's not necessarily true that $H \trianglelefteq G$.
This immediately gives an example in the topological setting: the class of topological spaces and covering maps is a category, but the class of topological spaces and normal (i.e., regular or Galois) coverings is not a category since it doesn't satisfy composition.
Are there other natural examples in the topological setting? The class of constant maps is not a category since it doesn't satisfy the identity axiom, but I'm more interested in failures of the composition property. I'd also be curious about algebraic examples if folks can't come up with interesting topological ones.