Not terribly much believe it or not. Algorithms and data structures is a very broad field. Obviously if you want to delve into some kinds of algorithms in particular, you need to know much more math. Generally though, for a good basic grasp of algorithms, you mostly need good problem solving abilities and a broad exposure (not necessarily too deep) to various kinds of mathematics. Number theory definitely is helpful as is some basic graph theory. If you really want to be able to analyze algorithms, you'll also need a fairly solid grasp of Big O notation.
This book is great for getting exposed to many various branches of math through problem solving:
Most algorithm books should at least gloss over Big O notation.
If you're interested in going deeper, you can use Knuth's Concrete Mathematics.