How to find the limit of $a_n=\int_{\mathcal{D}}(3xy-x^2y-xy^2)^n\lambda_2(dx dy)$ on $\mathcal{D}=\{(x,y)\in\mathbb{R}^2|x\gt0,y\gt0,x+y\lt3\}$ Given the triangular area  $$\mathcal{D}=\{(x,y)\in\mathbb{R}^2|x\gt0,y\gt0,x+y\lt3\}$$
and the sequence $(a_n)_{n\in\mathbb{N}}$ with $$a_n=\int_{\mathcal{D}}(3xy-x^2y-xy^2)^n\lambda_2(dx dy)$$
The notation $\lambda_2(dx dy)$ means, that we integrate  on 2-dim Lebesgue Measure $\lambda_2$.
Questions:
a) Find $b\in\mathbb{R}$ and $c\gt0$, such that
$$\mathscr{l}_n:=a_nn^bc^n\overset{n\rightarrow\infty}{\longrightarrow}{r\in\mathbb{R}^+}$$
b) Find the value of $r$.
Part of my idea:
Use affine-linear transformation$$\tilde{x}=n^\beta(x-1)\quad\tilde{y}=n^\beta(y-1)$$for appropriate $\beta\in\mathbb{R}$, then observe the asymptotics of the integral after transformation.
Define $\operatorname{f}:\mathbb{R}^2\rightarrow\mathbb{R}^2$, $(x,y)\mapsto(\frac{\tilde{x}}{n^\beta}+1,\frac{\tilde{y}}{n^\beta}+1)$
Jacobian $|\det D_f|=\left|\det\begin{pmatrix}
\frac{1}{n^\beta} & 0 \\0 & \frac{1}{n^\beta}\\
\end{pmatrix}\right|$=$n^{-2\beta}\gt0,\forall{n}\in\mathbb{N}$
Then $$a_n=n^{-2\beta}\int_\mathcal{D}(3(\frac{\tilde{x}}{n^\beta}+1)(\frac{\tilde{y}}{n^\beta}+1)-(\frac{\tilde{x}}{n^\beta}+1)^2(\frac{\tilde{y}}{n^\beta}+1)-(\frac{\tilde{x}}{n^\beta}+1)(\frac{\tilde{y}}{n^\beta}+1)^2)^nd\lambda_2(x,y)\quad=n^{-2\beta}\int_{\mathcal{D}}(\frac{-\tilde{x}^2n^{\beta}+n^{3\beta}-\tilde{x}^2\tilde{y}-\tilde{x}\tilde{y}^2-n^{\beta}\tilde{y}^2+n^{\beta}\tilde{x}\tilde{y}}{n^{3\beta}})^nd\lambda_2(\tilde{x},\tilde{y})$$
and that's where i get stuck. How do i proceed with the calculation?
Thanks in advance!
 A: Note that
$$\begin{align}
a_n&=\iint_{\mathcal{D}}(xy)^n  (3-x - y)^n dx dy\\
&=\iint_{\mathcal{D}}(xy)^n\left(\int_{z=0}^{3-x-y}nz^{n-1}dz\right) dxdy\\
&=n\int_{x=0}^3\int_{y=0}^{3-x}\int_{z=0}^{3-x-y}x^ny^nz^{n-1} dxdydz\\
&=n3^{3n+2}\int_{X=0}^1\int_{Y=0}^{1-X}\int_{Z=0}^{1-X-Y}X^nY^nZ^{n-1}dXdYdZ
\end{align}$$
where we applied the substitutions $x=3X$, $y=3Y$, $z=3Z$.
Then, from Help evaluating triple integral over tetrahedron , we find
$$a_n=3^{3n+2} \frac{(n!)^3}{(3n+2)!} 
\sim \frac{2\pi}{\sqrt{3}}\cdot\frac{1}{n}$$
where at the last step we used the Stirling's approximation.
Check the formula of $a_n$ at WolframAlpha.
Therefore $b=c=1$ and $r= \frac{2\pi}{\sqrt{3}}$.
A: As a corollary of the beta integral formula, you have for nonnegative integers $a$ and $b$ that
$$\int_0^1 x^a (1-x)^b \,dx = \frac{a! b!}{(a + b + 1)!}.$$
Now as a further corollary of this, let us fix a constant $C > 0$; then the substitution $x = Ct$ will give
$$\int_0^C x^a (C-x)^b \,dx = C \int_0^1 (Ct)^a [C(1-t)]^b\,dt = C^{a+b+1} \int_0^1 t^a (1-t)^b\,dt = \frac{C^{a+b+1} a! b!}{(a+b+1)!}.$$
Therefore, in the desired integral, let us evaluate the integral with respect to $y$ first: by the above formula, $$\int_0^{3-x} x^n y^n(3-x-y)^n\,dy = x^n \frac{(3-x)^{2n+1} n! n!}{(2n+1)!}.$$
And then, $a_n$ is the integral of this from $x=0$ to $x=3$:
$$a_n = \int_0^3 \frac{x^n (3-x)^{2n+1} n! n!}{(2n+1)!} \,dx = \frac{n! n!}{(2n+1)!} \cdot 3^{3n+2} \frac{n! (2n+1)!}{(3n+2)!} = 3^{3n+2} \frac{(n!)^3}{(3n+2)!}.$$
From here, as in the answer by Robert Z, you can use Stirling's formula to conclude that $a_n\sim \frac{2\pi}{n\sqrt{3}}$ as $n\to\infty$.
