Evaluate the double integral. 
To find:

$$\iint_R(x^2-xy)dA$$

enclosed by

$$y=x, y=3x-x^2,$$
$$x=3x-x^2,$$
$$x=0, x=2,$$
$$y=0, y=2,$$
$$R=((x,y): 0\le x\le 2), (x \le y \le 3x-x^2)$$
$$\int_0^2\int_x^{3x-x^2}(x^2-xy)\,dy\,dx$$
$$\int_0^2 \left. \left(x^2y-x\frac{y^2}{2}\right) \right|_x^{3x-x^2}\,dx$$

where to from here?

 A: Given your domain, let's first make a small correction:
\begin{align}
R &= \left\{(x,y) \in \mathbb{R}^2: 0 \leq x\leq 2, x \leq y \leq 3x-x^2 \wedge 2  \right\} \\
&= \left\{(x,y) \in \mathbb{R}^2: 0 \leq x\leq 1, x \leq y \leq 3x-x^2   \right\} \\
&\qquad\cup \left\{(x,y) \in \mathbb{R}^2: 1 < x \leq 2, x \leq y \leq 2   \right\}
\end{align}
So,
\begin{align}
I
&= \int_R x^2 - xy \;\mathrm{d}A\\
& = \int_0^2 \int_x^{3x-x^2 \wedge 2} x^2 - xy \;\mathrm{d}y\,\mathrm{d}x \\
%
%
&= \int_0^1 \int_x^{3x-x^2} x^2 - xy \;\mathrm{d}y\,\mathrm{d}x +
   \int_1^2 \int_x^{2} x^2 - xy      \;\mathrm{d}y\,\mathrm{d}x \\
%
%
&= \int_0^1 x \left[x y - \frac{y^2}{2} \right]_{x}^{3 x-x^2} \;\mathrm{d}x +
   \int_1^2 x \left[x y - \frac{y^2}{2} \right]_{x}^{2}       \; \mathrm{d}x \\
%
%
&= \int_0^1 \left(- 2 x^3 + 2 x^4 -\frac{x^5}{2} \right) \;\mathrm{d}x +
   \int_1^2 \left(-2 x +2 x^2 -\frac{x^3}{2}  \right)   \;\mathrm{d}x \\
%
%
&= \left[-\frac{x^4}{2}+\frac{2 x^5}{5}-\frac{x^6}{12} \right]_0^1 +
   \left[-x+\frac{2 x^3}{3}-\frac{x^4}{8} \right]_1^2 \\
%
%
&= \left(-\frac{11}{60} \right) +\left(-\frac{5}{24}\right)\\
%
%
&= -\frac{47}{120}
\end{align}
A: With the change of variables $u=x$ and $v=y-x$, that is $x=u$ and $y=v-u$,  we have the Jacobian $\left|\frac{\partial(x,y)}{\partial (u,v)}\right|=1$; from $0\le x\le 2$ we have $0\le u\le 2$ and from $x\le y\le 3x-x^2$, we have $0\le y-x\le 2x-x^2$ we have $0\le v\le 2u-u^2$. The integral becomes
\begin{align}
I&=\int_0^2\int_0^{(2-u)u} (-uv)\,\mathrm dv\,\mathrm du\\
&=\int_0^2 (-u)\,\frac{u^2(2-u)^2}{2} \mathrm du=-\frac{1}{2}\int_0^2(4u^3-4u^4+u^5)\,\mathrm du\\
&=\left.-\frac{1}{2}\left(4u^4-\frac{4u^5}{5}+\frac{u^6}{6}\right)\right|_0^2=-\frac{2^4}{2}\left(4-\frac{4\cdot 2}{5}+\frac{2^2}{6}\right)=-\frac{8}{15}
\end{align}
Check with WolframAlpha
