Find the flux of the vector field $ \ F(x,y,z)=x \hat i+y \hat j-z \hat k \ $ Find the flux of the vector field $ \ F(x,y,z)=x \hat i+y \hat j-z \hat k \ $ through the portion of the plane in the first octant with intercept $ \ (2,0,0) , \ (0,3,0) , \ (0,0,6) \ $ , where positive flow is defined to be in positive z-direction.
Answer:
$ \boldsymbol \nabla \times F =0 $
Thus,
$$ \mathrm{Flux} =\iint_S (\boldsymbol \nabla \times F)\cdot \widehat n \,dS  =0 $$
But the answer should be non-zero and would be any one option among$ \ (i) \ 5/2 , \ (ii) \ -5/4 , \ (iii) \ 160/3, \ (iv) \ 205/7 \ $
which option is correct?
 A: First we find an equation for the plane. Recall that a plane passing through $(a, 0, 0)$, $(0, b, 0)$, $(0, 0, c)$ has the general form
$$\frac{x}{a} + \frac{y}{b} + \frac{z}{c} = 1$$
Whence the plane here is
$$\frac{x}{2} + \frac{y}{3} + \frac{z}{6} = 1$$
Clearing fractions we get $3x + 2y + z = 6$ or $z = 6 - 3x - 2y$.
From the plane we have two normal vectors to choose: $\mathbf{n} = \left(-3, -2, -1\right)$ or $\mathbf{n_1} = (3, 2, 1)$
These are NOT unit normal vectors, but recall that the radicals will cancel away. We just need to ensure there is $1$ or $-1$ in the $z$ position.
Let's choose $\mathbf{n_1}$.
We can not find $\mathbf{F}\cdot \mathbf{n_1}$, and we can write $\mathbf{F}$ in terms of $x, y$ cutting $z$ with the above expression:
$$\mathbf{F}(x, y) = ( x, y, -z ) = (x, y, -(6-3x-2y))$$
Thus
$$\mathbf{F}\cdot\mathbf{n_1} = (x, y, -6 +3x + 2y)\cdot (3, 2, 1) = 3x + 2y -6 + 3x + 2y = 6x + 4y - 6$$
This will be the integrand.
The bounds of integration lie in the $XY$ plane, hence you just need to find the bounds for the integration like $0 < x < a$ and $0< y < g(x)$ and calculate the integral.
In this case we have the bounds as $0 < x < 2$ and $0 < y < 3 - 3/2 x$ whence 
$$\int_0^2 \int_0^{3 - 3/2 x} 6x + 4y - 6\ \text{d}y\ \text{d}x = 6$$
So, either I made a mistake (which is very possible since I'm answering from the bus) or none of the answers above.
