Application of Poincaré-Bendixson theorem Consider the system
$$x' = 3xy^2-x^2y \\
y' = 5x^2y - xy^2$$
Show that the system has no periodic solutions.
This is a tricky example. Linearization leads nowhere and I'm having a hard time constructing a Lyapunov function that does the trick. $V = 1/2(x^2+y^2)$ gives
$$V'(x,y) = 3x^2y^2-x^3y +5x^2y^2 -xy^3 = 8x^2y^2 -xy(x^2+y^2))$$
But this doesn't tell us much nice things about the origin. If anything, it looks as though the origin is repelling since small perturbations gives us that the $8x^2y^2$ term dominates the minus term. Maybe it's possible to show that there are no elliptical orbits somehow, but that doesn't exclude other, more exotic, periodic trajectories.
How to proceed...?
 A: The title is misleading: it is the Bendixson's criterion what must be used that states that for the system
$$\begin{align}
x'&=f(x,y)\\y'&=g(x,y)
\end{align}
$$
if $\frac{\partial f}{\partial x}+\frac{\partial g}{\partial y}\neq 0$
in a simply connected region $R$, then the system has no closed trajectory inside $R$. (See e.g. http://math.mit.edu/~jorloff/suppnotes/suppnotes03/lc.pdf for a proof)
Now, in your case, first of all, note that the axes are invariant, so no closed trajectory can touch one of them. Then, inside any of the quadrants, compute:
$$\begin{aligned}
\frac{\partial f}{\partial x}+\frac{\partial g}{\partial y} &= 3y^2-2xy+5x^2-2xy =3\left(y^2-\frac{4}{3}xy+\frac{5}{3}x^2\right) \\
&= 3 \left(\left(y-\frac{2}{3}x\right)^2+\frac{5}{2}x^2\right)>0
\end{aligned}$$
since any (strict) quadrant excludes the origin, and the conclusion results from Bendixson's criterion.
A: I'll give the solution to your non-linear system $$\frac{dx}{dt}=3xy^2-x^2y$$
and $$\frac{dy}{dt}=5x^2y-xy^2$$
Divivding eq 2 by eq 1, we get $$\frac{dy}{dx}=\frac{5x^2y-xy^2}{3xy^2-x^2y}$$
Now, dividing by $x^3$ both the numenator and the denumenator we get that $$\frac{dy}{dx}=\frac{5\frac{y}{x}-(\frac{y}{x})^2}{3(\frac{y}{x})^2-\frac{y}{x}}$$
Let $y=x \cdot v$ then $\frac{dy}{dx}=x\frac{dv}{dx}+v$, thus the D.E transformed into a separable eq.
$$x\frac{dv}{dx}+v=\frac{5v-v^2}{3v^2-v}=\frac{5-v}{3v-1},$$ therefore
$$x\frac{dv}{dx}=\frac{5-v-3v^2+v}{3v-1}=\frac{5-3v^2}{3v-1}$$
$$ \Rightarrow \frac{3v-1}{5-3v^2}dv=\frac{dx}{x}$$ 
$$ \Rightarrow \int{\frac{3v-1}{5-3v^2}dv}=\int{\frac{dx}{x}}$$ 
Setting $
\sqrt {\frac{3}{5}} v= \sin \theta   \Rightarrow \sqrt {\frac{5}{3}} \cos \theta d\theta  = dv$, then 
$$\int{\frac{3\sqrt {\frac{5}{3}} \sin \theta 
-1}{5(1- \sin^2 \theta) }\sqrt {\frac{5}{3}} \cos \theta d\theta}=\int{\frac{dx}{x}}$$ 
so that $$\int{\frac{3\sqrt {\frac{5}{3}} \sin \theta 
-1}{5(1- \sin^2 \theta) }\sqrt {\frac{5}{3}} \cos \theta d\theta}=\int {\tan \theta d\theta }  - \sqrt {\frac{5}{3}} \frac{1}{5}\int {\sec \theta d\theta }  
\\
=  - \ln \left| {\cos \theta } \right| - \frac{1}{{\sqrt {15} }}\ln \left| {\sec \theta  + \tan \theta } \right|
$$
Hence the solution is, $$
 - \ln \left| {\cos \theta } \right| - \frac{1}{{\sqrt {15} }}\ln \left| {\sec \theta  + \tan \theta } \right|
=\ln \left| {x } \right|  +C$$ Finally, we may write the result as
$$\left| {\cos \theta } \right|\cdot
\left| {\sec \theta  + \tan \theta } \right|^{{\textstyle{1 \over {\sqrt {15} }}}} 
=\frac{k}{x}$$ where $k=\exp(-C)$.
