Second degree linear derivate with exponential function, which general solution? Suppose $y''-4y=xe^{2x}$.
Solution of the homogenous equation is $y_H(x) = C_{1} e^{2x} + C_{2} e^{-2}$, after solving the characteristic equation with the guess $y=e^{rx}$.
Now my instructor insists that the one solution is either
$y(x) = Ax^{2} e^{2x} + Bxe^{2x}$
or
$y(x) = Ax^{2}e^{2x} + Bxe^{2x}+Ce^{2x}$
...and then I get the solution $y(x) = y_{H}(x) +\frac{1}{8}x^{2}e^{2x}-\frac{1}{16}xe^{2x}$
but I cannot understand the choices. Why do just the forms work? I have tried many different forms there and get lost/wasted a lot time -- and my instructor just says that it is a "guess" and smiles. How can I find the one solution without guessing? For one solution, things such as $y(x)=e^{2x}$ or $y(x)=xe^{2x}$ work so I am a bit lost why the more complex form is "the solution" and why it provides all solutions.
For example, the case $y=e^{2x}$. We get $y'=2e^{2x}$ and $y''=4e^{2x}$ so 
$y''-4y = 4e^{2x} - 4 e^{2x} = 0$
so $RHS = xe^{2x} = 0$ so $x=0$, a solution?! If not why? Ok it is naive, it does not contain all cases but how can I know that some form contains all?
 A: I believe this can be done without guesswork.
One way to get this mechanically is to use the method of Laplace Transforms.
Of course, I haven't tried out it out myself, but given that
$$\mathcal{L}[t^n e^{at}] = \frac{n!}{(s-a)^{n+1}}$$
I believe the approach will work for your problem.
A: Another way to do this is using the Exponential Shift Theorem (see e.g. http://www.math.ubc.ca/~israel/m215/coco/coco.html): For any polynomial $P$, constant $k$ and function $u$, $P(D) e^{kx} u = \exp(k x) P(D+k) u$ (where $D$ stands for derivative).  In your case $P(t) = t^2 - 4$, $k=2$, 
and $P(t+2) = (t+2)^2 - 4 = t^2 + 4 t$ so if $y = e^{2x} u$
we have $y'' - 4 y = e^{2x} (u'' + 4 u')$.  Now you want $y'' - 4 y = x e^{2x}$
so $u'' + 4 u' = x$.  Writing $v = u'$, we have the first-order linear equation in $v$: $v' + 4 v = x$, which has a solution $v = \frac{x}{4} - \frac{1}{16}$.  An antiderivative of this is $u = \frac{x^2}{8} - \frac{x}{16}$, corresponding to the particular solution $y = e^{2x} \left(\frac{x^2}{8} - \frac{x}{16}\right)$ of your original equation.  
A: The "right" approach is highly context-dependent.  What follows is from the perspective of an introduction to differential equations attached to a first-year calculus course.  
We describe an informal procedure, in which we make a plausible guess as to $y$, and test how well the guess works by calculating $y''-4y$ for this guess.  We then use the information obtained from the calculation of $y''-4y$ to improve our guess.  Of course more formal procedures are available, but it is nice to see how far one can get by playing a little. 
A very reasonable "guess" for a particular solution is $y=xe^{2x}$.  Let's check whether it works.  If $y=xe^{2x}$, then $y'=2xe^{2x}+e^{2x}$, and $y''=4xe^{2x}+4e^{2x}$.  Then $$y''-4y=4e^{2x}$$
But $4e^{2x}$ is definitely not the same function as $xe^{2x}$.  And multiplying the guess $xe^{2x}$ by a constant $k$ will do nothing useful.
Note that things would have worked out nicely if the differential equation had, for example,  $y''-3y$ on the left instead of $y''-4y$.  But the $-4y$ is exactly the right thing to make the $xe^{2x}$ term disappear. When the person posing the problem chose $-4$, (s)he chose the only constant that would make our lives difficult. Nasty!
However, this gives us an idea.  If we try $y=x^2e^{2x}$, and calculate $y''-4y$, maybe the $x^2e^{2x}$ term will disappear, which is exactly what we want.  Calculate.  If $y=x^2e^{2x}$, then $y'=2x^2e^{2x}+ 2xe^{2x}$ and $y''=4x^2e^{2x}+8xe^{2x} +2e^{2x}$.
Thus
$$y''-4y=8xe^{2x} +2e^{2x}$$
Close!  We should clearly multiply our guess by $(1/8)$ to get the $xe^{2x}$ term right.  And we will not be quite there, since then there will still be an unwanted  $(1/4)e^{2x}$ term in $y''-4y$ to get rid of.
Look back on the work we did at the beginning with $y=xe^{2x}$.  Then $y''-4y$ turned out to be $4e^{2x}$.  So to get rid of the unwanted $(1/4)e^{2x}$, we can simply add 
$(-1/16)xe^{2x}$ to the "guess" $(1/8)x^2e^{2x}$.
So we end up with the particular solution
$$y =\frac{x^2e^{2x}}{8}-\frac{xe^{2x}}{16}$$
A: Perhaps it would help to see the general method.
You want to solve an equation $Ly=x^n e^{\lambda x}$, where $L$ is a constant linear differential operator - in your case $Ly=y''-4y$. Let us solve $Ly=e^{(\lambda+\epsilon) x}$ instead: if we substitute $y=c(\epsilon)e^{(\lambda+\epsilon) x}$, we get $c(\epsilon)p(\lambda+\epsilon)=1$ (where $p$ is the characteristic polynomial of $L$, in your case $p(t)=t^2-4$), i.e. $c(\epsilon)=1/p(\lambda+\epsilon)$. 
We now expand
$$L\frac{e^{(\lambda+\epsilon) x}}{p(\lambda+\epsilon)}=e^{(\lambda+\epsilon) x}$$
to a power series in $\epsilon$, look at the term at $\epsilon^n$, and get a solution of $Ly=x^n e^{\lambda x}/n!$.
In your case $\lambda=2$, $p(t)=t^2-4$, and $c(\epsilon)=1/(\epsilon(4+\epsilon))$. We thus get
$$L\bigl((\epsilon^{-1}4^{-1}+(4^{-1}x-4^{-2})+\epsilon(4^{-1}x^2/2-4^{-2}x+4^{-3}) +\dots )e^{2 x}\bigr)=$$
$$=(1+\epsilon x+\epsilon^2 x^2/2 +\dots)e^{2 x}.$$
Looking at the term at $\epsilon$, we get
$$L((4^{-1}x^2/2-4^{-2}x+4^{-3})e^{2 x})=xe^{2 x},$$
i.e. $y=(4^{-1}x^2/2-4^{-2}x+4^{-3})e^{2 x}$ is a solution of your equation (we can drop $4^{-3}e^{2 x}$ from the solution, as it solves the homogeneous equation). 
