You lose surprisingly little. ZF with the axiom of infinity replaced by its negation turns out to be the set of hereditarily finite sets and you can still do calculus perfectly well.
See If all sets were finite, how could the real numbers be defined? for more. See also https://en.wikipedia.org/wiki/Hereditarily_finite_set.
On choice, if you simply remove the axiom, there is no hope of proving anything more than you could without it. But if you add the negation, then Ackermann's bijection provides a choice function. That said, I don't know whether it can be proven to be a well-defined choice function from within that axiom system. I suspect not.
Of more historic and philosophical interest is removing the law of the excluded middle. This gives you Constructivism. It turns out that you can still construct the real numbers, do calculus, and so on. However you get odd consequences such as numbers not always being comparable and all well-defined functions having to be continuous.