solve $y''-2y'-y=\sin{3x}$ $$y''-2y'-y=\sin{3x}, \quad y = y(x).$$
We find $y_h = Ae^{(1-\sqrt{2})x}+Be^{(1+\sqrt{2})x}$
However I have trouble with finding $y_p$.
Using the helping function:
$$ u''-2u'-u = e^{\mathrm{i} \, 3x}, \quad \mathrm{i}^2 = -1,$$ and then $$u = e^{\mathrm{i} \, 3x}z$$
I get $z''-2z'+6 \mathrm{i} \, z'-z-6 \mathrm{i} \, z-8=0$,  which I believe is clearly wrong and too complex.
The solution should be $$y = Ae^{(1-\sqrt{2})x}+Be^{(1+\sqrt{2})x} + \frac{1}{68}\left(3\cos{3x} - 5 \sin{3x} \right)$$
 A: The idea is to look for a particular solution of the differential equation
$$y''-2y'-y=e^{3ix}\tag E$$
on the form
$$y_p(x)=ae^{3ix}$$
and then take its imaginary part. Substitute $y_p $ in $(\rm E)$ gives
$$-9a-6ia-a=1\iff a=-\frac{1}{10+6i}=\frac{6i-10}{136}$$
so the desired particular solution for the given DE is
$$y_p(x)=\frac1{68}(3\cos(3x)-5\sin(3x))$$
A: Best way to solve is the method of undetermined coefficients. Since we know that the right-hand side is a $\sin(3x)$, the particular solution should include all forms of $\sin$ and $\cos$ with the same inside ($3x$).
Hence, I will assume:
$$y_p=A\sin(3x)+B\cos(3x)$$
$$y_p'=3A\cos(3x)-3B\sin(3x)$$
$$y_p''=-9A\sin(3x)-9B\cos(3x)$$
Plug into the differential equation:
$$\left(-9A\sin(3x)-9B\cos(3x) \right) -2\left(3A\cos(3x)-3B\sin(3x)\right)-\left(A\sin(3x)+B\cos(3x)\right)=\sin(3x)$$
Now we are going to collect like terms: $\sin(3x),\cos(3x)$.
$$\sin(3x)(-9A+6B-A)=\sin(3x)(1) \implies -10A+6B=1$$
$$\cos(3x)(-9B-6A-B)=\cos(3x)(0) \implies -6A-10B=0$$
Solve this $2\times 2$ system of linear equations:
$$A=\frac{6B-1}{10}$$
$$\frac{-3}{5}(6B-1)-10B=0 \implies \frac{-18}{5}B-10B=-\frac{3}{5} \implies B=\frac{3}{68}$$
$$-10A-6 \times \frac{3}{5}=1 \implies A=-\frac{5}{68}$$
Taking this now-solved coefficients and plugging them into the assumed form above with give the particular solution for this case.
$$y_p=-\frac{5}{68}\sin(3x)+\frac{3}{68}\cos(3x)$$
Therefore, it follows (as you mentioned) that the total solution is the sum of the particular and homogeneous solutions:
$$y(t)=Ae^{(1-\sqrt{2})t}+Be^{(1+\sqrt{2})t}+\frac{1}{68}(3\cos(3x)-5\sin(3x))$$
A: The $z$ in your trial solution should be a constant, inserting this into the inhomogeneous equation gives
$$
(-9-6i-1)e^{3ix}z=e^{3ix}\iff -2(5+3i)z=1 \iff -2\cdot 34\cdot z=5-3i.
$$
A: I would personally follow one of the next two hints:
Hint 1: since the non-homogenous part of your ODE is a linear combination of sines and cosines, set $y_p(x) = \alpha \, \sin{3x} + \beta \, \cos{3x}$  and solve for the constants $\alpha$ and $\beta$ by comparing term by term.
Hint 2: use the variation of parameters method and set $y(x) = A(x) y_1(x)$, where $y_1 = e^{(1- \sqrt{2})x} $, for example. Substitute back in the original ODE and solve for $A$.
Cheers!
A: Once you've found the general solution all you need to do is find the particular solution and add the two together. In this case the particular solution takes the form $C\sin 3x + D\cos 3x$. Just substitute that into the original equation and equate coefficients and solve the resulting linear system. 
