it's related to some constraint
This is correct: the constraint is that the surface must enclose volume $V$. The test function $\varphi$ describes a variation of the surface: we compare the original parametrization $X:U\to\mathbb R^3$ again the perturbed $X+\epsilon\varphi $. We are interested in the first derivative with respect to $\epsilon$, evaluated at $\epsilon=0$. For the surface area this derivative turns out to be $\iint_U 2H\langle \varphi, N\rangle $, and for the volume it is simply $\iint_U \langle \varphi, N\rangle $. At a critical point the gradient of the objective (surface area) is a multiple of the gradient of the constraint (volume). Hence, $\iint_U (2H-\lambda)\langle \varphi, N\rangle =0$.
It is not surprising that only the normal component of $\varphi$ matters. It should be geometrically intuitive that variation in tangential directions has $o(\epsilon)$ effect on both surface area and the enclosed volume. For this reason, it is convenient to consider only normal variations: that is, $\varphi=\psi N$ where $\psi$ is a scalar function.
The Wikipedia article on CMC surfaces has a lot of references, some of them expository.