Consider a matrix A (containing elements that possess N as unit) that maps a vector b (containing elements that possess m/N as unit) to a vector c (containing elements that possess m as unit). What units do the singular values of A have? And why?
When you do a singular value decomposition, $A = U\Sigma V^*$, you can put the unit whereever you like. You could even divide the unit between $U$, $\Sigma$ and $V^*$. So your question doesn't really make sense mathematically. The most natural thing would probably be to put it in $\Sigma$ and then the singular values would have the same units as your matrix entries. But that is really just a choice.