The general theory is for compact or semisimple Lie algebras (so includes all $SO(p,q)$) and is called the theory of weights. Decent books are Fulton and Harris's Representation theory: a first course or Sepanski's Compact Lie groups. Though also a google search will bring up a ton of notes on this.
Things get more complicated than the $SO(3)$ case since, in general, there are multiple ladder operators (which are called root vectors" in the mathematics literature), which interact with each other.