First of all, I think that you may be confusing "sphere" with "ball". If you take a sphere a drill a hole through it, you get a cylinder, not a torus. But if you take a ball and drill a hole through it, you get a $3$-manifold with boundary whose boundary is a torus.
If you take a ball and drill two holes through it intersecting at the center of the sphere, you get a $3$-manifold whose boundary is a genus $3$ surface. To see this, imagine stretching one of the holes very much, so that what you have is essentially a donut with a hole drilled through it perpendicularly to its axis of rotational symmetry. There are three holes: one in the center, and two on opposite sides of the donut.
Doing the same thing with three holes intersecting at the center will leave you with two extra holes, so in other words with a genus $5$ surface.