Nonhomogeneous second order ODE 
Solve  $y''-2y'-3y=3te^{-t}$

Attempt: now the solution has two parts. Namely, $y=y_h+y_p$ where $y_h$ is the solution of the homogeneous equation and $y_p$ is the solution of the nonhomogeneous equation. $y_h$ is easy to find.
$$y_h=c_1e^{3t}+c_2e^{-t}$$
I am struggling with finding $y_p$. The way I set it up is:
$$y_p=(At+B)e^{-t}=Ate^{-t}+Be^{-t}$$
Since $Be^{-t}$ is also a solution for $y_h$ I multiply RHS of $y_p$ by $t$ and get:
$$y_p=t(At+B)e^{-t}=At^2e^{-t}+Bte^{-t}$$
Then, I find derivatives of $y_p$ and plug them into differential equation, simplify, and get:
$$2A-2B-2At^2-4At=-3t$$
Now, if this is correct how would I solve for A and B?
It is not homework. Thank you for your help.
 A: Using:
$$y_p=t(At+B)e^{-t}=At^2e^{-t}+Bte^{-t}$$
For $y''-2y'-3y=3te^{-t}$, we get:
$$(2 A - 4B - 8At)e^{-t} = 3t e^{-t}$$
So, equating sides, we get:
$$-8A = 3 \rightarrow A = -\frac{3}{8}$$
and we have:
$$2A - 4B = 0 \rightarrow B = \frac{1}{2} A \rightarrow B = -\frac{3}{16}$$
This agrees with WA 
Regards
A: You are on the right track as far as a guess for $y_p$; something went wrong with your algebra.  If we guess that $y_p = (b+c t)e^{-t}$, then plugging this into the diff eq'n gives us, after much simplification:
$$(-4 b+2 c -8 c t) e^{-t}$$
Equating this with your RHS gives $c=-3/8$ and $b=-3/16$.  Plugging this specific form back into the equation produces $3 t e^{-t}$, as expected.
A: You definitely made a mistake, that's what Mathematica shows  
In[27]:= ClearAll[A, B]
y = (A*t^2 + B*t)*E^-t;
y' = D[y, t];
y'' = D[y', t];
S = Simplify[y'' - 2 y' - 3 y];

In[64]:= S

Out[64]= E^-t (-4 B + A (2 - 8 t))

In[65]:= sol = Solve[-4 q + 2 p == 0 && -8 p == 3, {p, q}];

In[63]:= Simplify[S /. {A -> p /. sol[[1]], B -> q /. sol[[1]]}]

Out[63]= 3 E^-t t

A: For getting the particular solution of this inhomogeneous equation, you guess an expression for it with two undetermined coefficients A and B, however you may neglect some conditions. There is a systematic method to cope with those kinds of inhomogenous equation, which the general solutions of its associated homogeneous equations are known. 
As you posted it: the associated general solutions of its homogeneous equations are:
$$y_h=c_1e^{3t}+c_2e^{−t}$$
now supposing the particular solutions is of the form:
$$y_p=v_1(t)e^{3t} + v_2(t)e^{-t} $$
The Wronskian of this homogeneous equ. is:
$$W=\begin{vmatrix}
e^{3t} & e^{-t} \\
(e^{3t})' & (e^{-t})'
\end{vmatrix}=-4e^{2t}$$
denote the inhomogeneous term as $Q(t)$: $3te^{-t}=Q(t)$
the systematic method states that :(note the integral constants are not important.)
$$ v_1(t) = \int \frac{-y_2(t)Q(t)}{W} \mathrm{d}t =-\frac{3}{64}e^{-4t}(4t+1) \\
v_2(t)=\int \frac{y_1(t)Q(t)}{W} \mathrm{d}t = -\frac{3}{8}t^2$$
Therefore one of the particular solutions is :
$$y_p=-\frac{3}{64}e^{-t}(8t^2+4t+1)$$
