This is a quite different and unrelated use of the term "extension." Generally the ambient category is an abelian category (although generalizations are possible), and in this context an extension of an object $B$ by an object $A$ is an object $C$ fitting into a short exact sequence
$$0 \to A \xrightarrow{f} C \xrightarrow{g} B \to 0$$
meaning that $f$ is a monomorphism, $g$ is an epimorphism, and $\text{ker}(g) = \text{im}(f)$ (see the link for definitions).
A subcategory is then said to be closed under extensions if whenever $A$ and $B$ lie in that subcategory, then so does any extension $C$. For example, inside the abelian category of abelian groups, the following (full) subcategories are closed under extensions (some of these are nice little exercises):
- Finite abelian groups
- Finitely generated abelian groups
- Torsion abelian groups
- Torsion-free abelian groups
- Divisible abelian groups
However, the following subcategories are not closed under extensions:
- Cyclic abelian groups
- $p$-torsion abelian groups