A solution of the PDE $au_x+bu_y=-u$ vanishes identically under certain boundary condition It has bothering me for days. We learnt characteristic methods and I try to use but it doesn't work. The question is as above. Thank you.
I think of a solution but I think it is somewhere wrong:
1
 A: If $u$ assumes it's maximum in the interior of $D$, let's say, in the point $x_0$, we have that $u_x(x_0)=u_y(x)=0$, so as you have noted, $$-u(x_0)=a(x_0)u_x(x_0)+b(x_0)u_y(x_0)=0,$$
therefore $u(x)\le u(x_0)\le 0$ for all $x\in D$. Thus, let's assume that $u$ attain it's maximum in the point $x_0\in \partial D$.
Now, the characteristic equations says that, along the curves $$(x'(t),y'(t))=(a(x(t),y(t),b(x(t),y(t)),$$
$u$ must satisfies $$\frac{du(x(t),y(t))}{dt}=-u(x(t),y(t)),$$
whence, if we define $v(t)=u(x(t),y(t))$, we obtain that $v(t)=Ce^{-t}$, for some constant $C$. If $C\le 0$, we are done. If $C>0$, take $t_0$ such that $v(t_0)=x_0$.
By hypothesis, $(x'(t_0),y'(t_0))\cdot (x(t_0),y(t_0))>0$, therefore, there is $\delta>0$ small such that, if $t<t_0$ and $|t-t_0|<\delta$, we have that $(x(t),y(t))$ belongs to the interior of $D$. 
However, once $C>0$, the function $v$ must be decreasing, which implies that for all such $t$ as above, $v(t)>v(t_0)$, which is an absurd, thus $C\le 0$ and $u\le 0$ in $D$.
Analogous arguments applies, when considering the minimum of $u$.
