Given a general region, find the double integral bounded between $y = x$ and $y=3x-x^2 $ $$ J  = \iint_R (x^2-xy)\,dx \,dy, $$
Suppose region R is bounded between $y = x$ and $y=3x-x^2 $
My attempt using vertical integration:
$$ \int^{x=2}_{x=0} \int^{y=3x-x^2}_{y=x} \left({x^2-xy}\right)dy\ dx$$
$$\int^2_0 \left[x^2y-x\frac{y^2}{2}\right]^{3x-x^2}_{x}\, dx$$
$$\int^2_0 \frac{-x^5+4x^4-4x^3}{2} \,dx $$
$$\boxed{J = -\frac{8}{15}}$$

My attempt using horizontal integration :
$$ \int^{y=2}_{y=0} \int^{x=y}_{x=3\,\pm \sqrt{9-y}} \left({x^2-xy}\right)dx\ dy$$
For $ x = 3+\sqrt{9-y}$
$$  \int^2_0 \left[\frac{x^3}{3}-\frac{x^2}{2}y\right]^y_{3+\sqrt{9-y} }\,dy$$
For $ x = 3-\sqrt{9-y}$
$$  \int^2_0 \left[\frac{x^3}{3}-\frac{x^2}{2}y\right]^y_{3-\sqrt{9-y} }\,dy$$

My doubts :
1.) How do I set my limit of integration for horizontal integration, if there is $\pm$ to be considered ?
2.) the answer as negative what does that imply in questions related to double integrals?
Could you guys please help
 A: $y=3x-x^2 \implies y = - (x-\frac{3}{2})^2 + \frac{9}{4}$
So equation of parabola is $ \ 9 - 4y = (2x-3)^2 \ $ and its vertex is $\left(\frac{3}{2}, \frac{9}{4}\right)$.
Intersection of parabola and the line $y = x$ is $(2, 2)$. So you can see that it is below the vertex and to the right of the axis of symmetry.
So if you are doing horizontal integration,
for $0 \leq y \leq 2, $ $x$ is bound between parabola and on the right by the line $y = x$. But for $2 \leq y \leq \frac{9}{4}$, $x$ is bound between parabola at both ends.
Left of the axis of parabola is given by $2x - 3 = - \sqrt{9-4y}$ and right of it is given by $2x - 3 =  \sqrt{9-4y}$

So the integral should be
$\displaystyle \int_0^2 \int_{(3 - \sqrt{9-4y})/2}^y (x^2 - xy) \ dx \ dy \ \ $ +
$\displaystyle \int_2^{9/4} \int_{(3 - \sqrt{9-4y})/2}^{(3 + \sqrt{9-4y})/2} (x^2 - xy) \ dx \ dy \ \ $
$ = - \frac{47}{120} - \frac{17}{120} = - \frac{8}{15}$
which matches your answer using vertical integration.
A: Your first approach is correct, but your second approach is not. The possible values of $y$ lie in the interval $\left[0,\frac94\right]$, because $\frac94$ is the maximum of $3x-x^2$ when $x\in[0,2]$. When $y\in\left[0,\frac94\right]$, then (see the picture below):

*

*if $y\in[0,2]$, then the possible values of $x$ lie in $\left[\frac{3-\sqrt{9-4y}}2,y\right]$;

*if $y\in\left[2,\frac94\right]$, then the possible values of $x$ lie in $\left[\frac{3-\sqrt{9-4y}}2,\frac{3+\sqrt{9-4y}}2\right]$.

So, the answer is\begin{multline}\int_0^2\int_{\frac{3-\sqrt{9-4y}}2}^yx^2-xy\,\mathrm dy\,\mathrm dx+\int_2^{9/4}\int_{\frac{3-\sqrt{9-4y}}2}^{\frac{3+\sqrt{9-4y}}2}x^2-xy\,\mathrm dy\,\mathrm dx=\\=\int_0^2\frac{1}{12} \left(-2 y^3-6 y^2+\left(45-11 \sqrt{9-4 y}\right)  y+18 \left(\sqrt{9-4 y}-3\right)\right)\,\mathrm dy+\\+\int_2^{9/4}\frac{1}{6}(18-11 y)\sqrt{9-4 y}\,\mathrm dy=-\frac{47}{120}-\frac{17}{120}=-\frac8{15}.\end{multline}

A: Note that in your second case, your region is divided into two (under and above the green line). When you find the inverse functions of $y(x)$'s you have $$x(y) = y$$
for $y = x$ and $$x(y) = \frac{3 \pm \sqrt{9-4y}}{2}$$ (separate them into two at $x = 3/2)$ for $y = 3x-x^2$
Now, move your pen from left to right to notice what should be the bounds for integration over each region.

