Laplace equation on a disk with domain $\Omega:x^2+y^2I have the given functional:
$$J(z)=\iint_\Omega\big((z_x)^2+(z_y)^2\big)dxdy$$
defined on the circular domain $$\Omega:x^2+y^2<R^2$$
with initial conditions $\iint_\Omega z^2dxdy=1$
where $z=z(x,y)$ and Dirichlet conditions apply by $z=0$ on $\partial \Omega$. So this is a bounded circle, however  applying the formula $$F_z-\frac{\partial}{\partial x}F_{z_x}- \frac{\partial}{\partial y}F_{z_y}=0$$  on $F(z,z_x,z_y)$ gives a Laplace equation on a rectangle:
$$z_{xx}+z_{yy}=\lambda z $$
Option 1
By the conditions, and "ignoring" the circle domain, I get the ansatz:
$$u(x,y)=A_{m,n}\sin\frac{n\pi}{L}x\sin\frac{m\pi}{L}y$$
which is easily solvable  for the constant $A_{m,n}$ by using the initial conditions.
But then, what does
$$u(x,y)=A_{m,n}\sin\frac{n\pi}{L}x\sin\frac{m\pi}{L}y$$ have to do with a circular domain?
Option 2
So I ignore the $x,y$ and go over to Laplace equation on a circular domain, with ansatz $$u(r,\theta)=u(r)\sin\frac{n\pi}{\alpha}\theta$$
where $\alpha=2\pi$, so getting the Bessel equation:
$$u_{rr}+\frac{1}{r}u_r+\bigg(\lambda-\frac{\big(\frac{n}{2}\big)^2}{r^2}\bigg)u(r)=0$$
which gives the solution:
$$u(r,\theta)=J_{n/2}\bigg(\frac{\alpha_{n,k}}{J_{n/2}(R)}r\bigg)\sin\frac{n}{2}\theta$$
where $\alpha_{n,k}$ are the Bessel zeros. However, this has been found without using the initial condition.
Option 3:
We find that a reasonable approach to this problem is to solve the PDE:
$$u_{rr}+\frac{1}{r}u_r+\frac{1}{r^2}u_{\theta\theta}=-\lambda u$$
as given in  option 2, however the solution of the angular function  $\Theta$ is (correctly as JackR suggested), not related to the ansatz given in the option 2. Instead, we have:
$$\frac{r^2R''+rR'+\lambda r^2R}{R}+\frac{\Theta''}{\Theta}=0$$
The two terms on the left must be a constant, since it is the only point they will intersect, so we have:
$$\frac{\Theta''}{\Theta}=-\mu$$
and
$$r^2R''+rR'+\lambda r^2R=0$$
The former we solve as such:
Since boundary conditions on $\Theta$ are $2\pi$-periodic, so then $\mu=n^2$ and
$$\Theta(\theta)=\cos n\theta+\sin n\theta$$
Using $\mu = n^2$, equation for the radial part $R(r)$ we get
$$r^2R′′ + rR′ + (\lambda r^2 − n^2)R = 0$$
Which is the Bessel equation. Here we have to consider two solutions:
$\lambda=0, \lambda>0$
The first, $\lambda=0$:
$$J(r)=ar^{\mu}+br^{-\mu}$$
Since the circle is bounded and zero at the periphery, $b=0$, so with I.C. we would get the solution:
$$u(r,\theta)=ar^{\mu}(\sin \theta+\cos \theta)$$
The we have to convert this back to Cartesian coordinates and get:
$$u(x,y)=a\bigg(\frac{x}{\cos\theta}\bigg)^{\mu}\bigg(\frac{x}{r}+\frac{y}{r}\bigg)$$
But this, and also the case for $\lambda>0$ do not give solutions without the $r$-variable, which is incompatible with the aditional condition that $\int\int_\Omega z^2 dxdy=1$
So what is the right approach here?
Thanks
 A: You are correct that the Euler-Lagrange equation associated with minimising the functional $J$ subject to the constraint $\int_\Omega u^2 \,dx =1$ is the eigenvalue problem $$\frac{\partial^2 u}{\partial x^2}+\frac{\partial^2 u}{\partial y^2}=\lambda u  \qquad \text{in } \Omega. $$ However, your ansatz is wrong - you cannot just 'ignore' that the domain is circular because you need to apply the boundary condition $u=0$ on $\partial \Omega$ which you won't be able to do if you assume $u(x,y) = X(x)Y(y)$ (i.e. use rectangular coordinates).
Instead, you need to use polar coordinates, i.e make the guess $u(x,y) = R(r) \Theta(\theta)$. You have kind of done this correctly in the second half of your question, but there is no reason that $\Theta(\theta) = \sin (n \pi \theta/\alpha)$. This is actually a very common problem in courses on separation of variables and Sturm-Liouville theory, so if you search "Dirichlet Eigenvalues on a Disk" you should be able to find some resources online which give the answer in a lot of detail.
A: What you refer to as option 2 contains the right idea towards a solution. To find a solution write the function $z(r,\theta)=\sum_{n=-\infty}^\infty f_n(r)e^{in\theta}$ and then show using the PDE you derived from the functional + constraint that they must satisfy the Bessel equation and therefore
$$f_{n}(r)=c_nJ_n(\sqrt{\lambda}r)$$
Now, in order to impose the boundary condition at the boundary of the disk, one must impose $c_n=0$ for all $n$ except for one, for which the Bessel function argument must be at a zero of the function at $r=R$. Then, we see that there is a family of solutions labeled by two parameters given by
$$f_{n,k}(r)=c_nJ_n\left(\frac{\mu_{n,k}r}{R}\right)\cos(n\theta)~~,~~~~~ n,~k-1\in \mathbb{N}$$
where $\mu_{n,k}$ stands for the $k$-th zero of the Bessel function of order $n$.
The normalization condition $\int_{\Omega}z^2 d^2 x=1$  determines the coefficients $c_n$ (hint: change to polar coordinates and use properties of the Bessel functions):
$$c_n=(\pi R^2J_{n+1}(\mu_{n,k})^2)^{-1/2}$$
There are many extremizers of the functional presented above with the boundary conditions included, but there is only one true minimizer. We find by numerical evaluation that the lowest order Bessel function at the lowest order zero
$$z_{\min}(r,\theta)=\frac{J_0\left(\frac{\mu_{0,1}r}{R}\right)}{\sqrt{\pi R^2 J_1(\mu_{0,1})^2}}$$
minimizes the functional.
Note: Changing to polar coordinates for the last step is also crucial:$\int_\Omega (z_x^2+z_y^2) dxdy=\int_0^R rdr\int_0^{2\pi} d\theta(z_r^2+z_\theta^2/r^2 )$. All integrals involving Bessel functions here can be done explicitly through other Bessel functions.
