There is a weaker notion than 'class of maps stable under pullback', and that is a coverage, which is a class of maps that are stable under weak pullback - this is like pullback but only the existence, not the universal property (i.e. being final in all cones). This is all you need to define sheaves. Note that you don't need to work with just epimorphisms, and you don't need your category to have any a priori limits.
Coverages tend to be given by very small amount of data, as their closures under all the usual operations (composition, isomorphisms, taking sieves) give rise to an equivalent site structure, but a coverage is the minimum you need to specify. For example, the category of manifolds has a coverage where the covering families are good open covers (every open is diffeomorphic to some $\mathbb{R}^n$, and so are all their finite intersections). This gives a site equivalent to the one where covering families are collections of jointly surjective submersions. The first is a lot less data!