What is the right domain for this Hamiltonian I want to define a proper domain $D(H) \subset L^2$ for this Hamiltonian ( $\theta$, $\phi$ are the standard angles in spherical coordinates). Furthermore, the wave function is supposed to satisfy that the function itself and its derivative is continuous at the boundary of the intervals of $\theta$ and $\phi$.)
$$   \left\{ -\frac{1}{\sin^2\theta} \left[
\sin\theta\frac{\partial}{\partial\theta} \Big(\sin\theta\frac{\partial}{\partial\theta}\Big)
+\frac{\partial^2}{\partial \phi^2}\right]\right\} \psi(\theta,\phi)+V(\theta)\psi(\theta,\phi) = E \psi(\theta,\phi),$$
where $V \in C^{\infty}([0,2\pi]) \rightarrow \mathbb{R}$.
One normally wants that this operator shall be self-adjoint (therefore it should be a dense domain). Could anybody explain to me how to find a proper domain for this operator?
 A: You have to better specify the boundary conditions, because you may have different self-adjoint extensions, depending on them (and you have to be sure your operator maps you in the same space with same boundary conditions).
I suppose that, once you have fixed the suitable conditions, $C^\infty([0,2\pi]\times [0,2\pi])$ should be a domain of essential self-adjointness. Furthermore, your operator is of the form $H=H_0+V$ with $V$ very regular, so you may apply Kato-Rellich theorem to have self-adjointness on the domain of $H_0$ (angular momentum operator).
A discussion of boundary conditions for a general class of operators that include yours can be found in this paper of Calkin.
A: Hilbert Space: You have not specified the underlying Hilbert space $X$, but I assume you want $X=L^{2}_{\sin\theta}([0,\pi]\times[0,2\pi])$ with weighted inner-product
$$
       (f,g) = \int_{0}^{2\pi}\int_{0}^{\pi}f(\theta,\phi)\overline{g(\theta,\phi)}\sin\theta\,d\theta\,d\phi.
$$
The operator
$$
           Lf = -\frac{1}{\sin\theta}\frac{\partial}{\partial\theta}\sin\theta\frac{\partial}{\partial\theta}f-\frac{1}{\sin^{2}\theta}\frac{\partial^{2}}{\partial\phi^{2}}f
$$
is of primary concern. I expect a selfadjoint domain for $L$ to also work for $L+V$ because $V$ will be relatively bounded for any reasonable domain. $L$ can be viewed as a restriction of the Laplacian.
Domain: Let $L_{0}$ be the formal operator $L$ restricted to $C^{\infty}$ functions on $[-1,1]\times[0,2\pi]$ which are periodic (along with all derivatives) in $\phi$, and which satisfy
$$
     f,\frac{\partial f}{\partial \theta}, \frac{1}{\sin\theta}\frac{\partial f}{\partial\phi},Lf \in X.
$$
This is a fairly natural domain because
$$
\begin{align}
 \sin\theta(Lf)\overline{f} & = -\left(\frac{\partial}{\partial\theta}\sin\theta\frac{\partial f}{\partial\theta}\right)\overline{f}-\frac{1}{\sin\theta}\frac{\partial^{2}f}{\partial\phi^{2}}\overline{f} \\
      & = -\frac{\partial}{\partial\theta}\left(\sin\theta\frac{\partial f}{\partial\theta}\overline{f}\right)+\sin\theta\left|\frac{\partial f}{\partial\theta}\right|^{2}
        -\frac{\partial}{\partial\phi}\left(\frac{1}{\sin\theta}\frac{\partial f}{\partial\phi}\overline{f}\right)+\frac{1}{\sin\theta}\left|\frac{\partial f}{\partial\phi}\right|^{2},
\end{align}
$$
which leads to
$$
    (L_{0}f,f) =\lim_{\epsilon\downarrow 0}\int_{0}^{2\pi}\int_{\epsilon}^{\pi-\epsilon}(Lf)\overline{f}\sin\theta\,d\theta\,d\phi
    = \left\|\frac{\partial f}{\partial\theta}\right\|^{2}
        +\left\|\frac{1}{\sin\theta}\frac{\partial f}{\partial\phi}\right\|^{2},\;\;
   f \in \mathcal{D}(L_{0}).
$$
It also follows that $(L_{0}f,g)=(f,L_{0}g)$ for all $f,g\in\mathcal{D}(L_{0})$.
There are reasons why this domain is the correct one for Quantum; for one thing, the operators $\frac{\partial}{\partial\theta}$ and $\frac{1}{\sin\theta}\frac{\partial}{\partial\phi}$ are defined, and these are useful operators related to momentum; these operators are needed to give a sum-of-squares factoring for the Hamiltonian. This domain is also suitable for the spherical harmonics. This domain is related to the Friedrich's extension, which is a compelling reason to consider it from the viewpoint of Mathematics and of Energy. Another important fact is that this domain eliminates the function
$$
         \delta_{0}(\theta)=\ln\left[\frac{1+\cos\theta}{1-\cos\theta}\right],
$$
which satisfies $L\delta_{0}=0$, even though $\delta_{0}\in X$, and $L\delta_{0}\in X$. No Quantum selfadjoint operator should have $\delta_{0}$ in the domain. There are boundedness conditions you can impose that will imply the above, but the $L^{2}$ conditions I stated can be imposed on the Sobolev space in a natural way, and these conditions give a complete description of a domain on which $L$ is selfadjoint in the strictest sense.
