To derive the representation of SO(3) one uses the ladder operator method. What is the theoretical basis for this method? Often the ladder operators are simply stated in the textbooks of quantum mechanics. Is there a way to derive the lowering and raising operators ? Can this method be generalized to SO(n), SO(n,k), ... ?
The general theory is for compact or semisimple Lie algebras (so includes all $SO(p,q)$) and is called the theory of weights. Decent books are Fulton and Harris's Representation theory: a first course or Sepanski's Compact Lie groups. Though also a google search will bring up a ton of notes on this.
Things get more complicated than the $SO(3)$ case since, in general, there are multiple ladder operators (which are called ``root vectors" in the mathematics literature), which interact with each other.