I am currently self-studying abstract algebra from Artin. In that background, I am looking for a problem book in a spirit somewhat similar to Problems in Mathematical Analysis by AMS so that I have a lot of problems to solve.
I don't know if I would call it a "problem book" but Herstein's fantastic "Topics in Algebra" has some amazing basic problems in abstract algebra. If you want more commutative algebra, Atiyah-Madonald have a lot too, but you should probably wait with that one.
For a problem book, I would recommend: Exercises in Algebra: A Collection of Exercises, in Algebra, Linear Algebra and Geometry (Algebra, Logic and Applications , Vol 6) .
Look for old quals in Algebra too and old exams. I can not, if you're learning abstract algebra, recommend Aluffi's book "Algebra: Chapter 0" enough. I used it as a first introduction and I fell in love with the subject. Good luck!
For a first pass through the material, I really enjoyed Pinter's A Book of Abstract Algebra, as it presented the material quite well (although it did not go very deep). Also, I started out hating Algebra: Pure and Applied by Papantonopoulou, but I actually quite enjoyed it by the end of the course I used it for. Tons of problems in that one.
While I think that doing all the problems in Artin is more than enough, you might want to look through the problems on algebra in the Berkeley Problems in Mathematics and see whether you can solve them reasonably quickly. The Berkeley Problems is not a book to look for fascinating algebra problems, though. Unfortunately, I don't know of any problem books in algebra. I think Artin actually has very good, not-so-standard problems for undergraduate algebra.