How find a general solution for this inhomogeneous $2^{nd}$-order DE: $y''-2y'+y=xe^{-x}\cos(x)$? Let the non-homogeneous differential equation:
$$\boxed{y''-2y'+y=x \cdot e^{-x} \cdot \cos(x) } \tag{1}$$
Notice that the RHS is of the form: $ b(x)=x\cdot e^{sx} \cdot \cos(tx) $ where $s=-1, t=1$

Theorem 1

Let $y_p(x)$ be a particular solution of the non-homogeneous D.E and $y_0(x)$ be the general solution of the associated homogeneous equation (a.k.a complementary equation) then the general solution of the non-homogeneous equation is:
$$ y_{g}(x) = y_p(x) + c_1 \cdot y_0(x)$$


Complementary D.E's partial solution
The complementary D.E is a second order linear D.E: $\boxed{y''-2y'+y=0 } \quad(2)$
It's characteristic equation is: $\lambda^2 - 2\lambda + 1 = 0 $ Notice that $\sqrt{b^2-4ac}=0$ therefore $\lambda= \frac{-b}{2a} = 1$
Hence, the general solution of the complementary D.E is: $\boxed{y_0(x) = c_1e^x+c_2xe^x } $

Homogeneous D.E's general solution
Method of undetermined coefficients
Theory suggests that

An educated guess about the particular solution of $(1)$ would be a solution of a similar form.


*

*My textbook suggests to use $y_{sub}(x) = z(x)e^{(s-it)x}$ as a guess to solve a small alternation of $(1)$ that is $\boxed{y''-2y'+y=P(x)e^{(s-it)x} \quad(4)}$ (where of course $e^{ix} =\cos(x)+i \sin(x)$). It then states that $Im(y_p(x))$ or $Re(y_p(x))$ will be the partial solution for $(1)$.


*Another resource suggests that for this form of $b(x)=ae^{αx} \cos βx+be^{αx} \sin βx$ I use $ Ae^{αx} \cos βx+Be^{αx} \sin βx $ as a guess with a note that "The guess must include both terms even if either  a=0  or  b=0".
Lets follow my textbook's advice:
$\text{Let}\quad \:\:\: y_{sub}(x) = z(x)\cdot e^{(-1+i)x} \\
\text{then} \quad \: y_{sub}^{(1)}(x) =(z'(x)-(1-i)z(x)) \cdot e^{(-1+i)x}  \\
\text{and} \quad \:\: y_{sub}^{(2)}(x) = (z''(x) - (2- 2i)z'(x) - 2iz(x))\cdot e^{(-1+i)x}$
Plugging the partial and its derivatives in $(4)$ we have:
$y''-2y'+y=x \cdot e^{(-1+i)x} \iff \\ $
$(z'' - (2- 2i)z' - 2iz)\cdot e^{(-1+i)x} + (-2z'+2(1-i)z) \cdot e^{(-1+i)x} + z\cdot e^{(-1+i)x} =  x \cdot e^{(1-i)x} \iff\\$
$z'' - (2- 2i)z' - 2iz -2z' + 2(1-i)z + z = x \iff  $
$z''-4z'+2iz'-4iz+3z=x \iff \\$
$$ \boxed{z''+2z(-2+i)+z(3-4i) = x} \tag{5}$$
Now, $(5)$ has $b(x) = $ polynomial therefore a partial solution guess would be of the form

*

*$y_{sub_2} = ax+b \iff$

*$y^{(1)}_{sub_2} = a \iff$

*$y^{(2)}_{sub_2} = 0$
Hence plugging the partial solution and the derivatives in $(5)$ we have:
$z''+2z(-2+i)+z(3-4i) = x \iff$
$0 +2a(-2+i)+(ax+b)(3-4i) -x =0 \iff $
$ -4a +i2a +3ax -i4ax +3b -i4b -x =0 \iff $
$$ \boxed{[(3a-1)x - (4a +3b)] + i[4ax + (2a -4b) ] = 0} \quad (6)$$
Let's take the imaginary part of $(6)$ then:
$$ (E) =
\left\{ 
\begin{array}{c}
(3a-1)x - (4a +3b) = 0 \\
4ax + (2a -4b) = 0
\end{array}
\right. 
$$
But there is no solution for this system... For example one the first equation $a = \frac13$ and on the second $a=0$ which is a contradiction. I triple checked the calculations (thus I reckon they are correct). I can't understand what went wrong. But most of all I can't really grasp the way of solving these inhomogeneous differential equations (that include a polynomial, an exponential and a trigonometric function). It all seems overly complicated to me.
I know this is a big post and not one, but several questions arise, so I for clarity purposes, I'll stick to the initial question: How to solve this differential equation?
P.S: Of course any other answers are highly appreciated :)
Cheers!
 A: Hint:
$$y''-2y'+y=xe^{-x}\cos(x)$$
This differential equation is equivalent to:
$$(ye^{-x})''=xe^{-2x} \cos x$$
Integrate twice.

Edit1:
$$y''-2y'+y=xe^{-x}\cos(x)$$
Multiply by $e^{-x}$
$$e^{-x}(y''-2y'+y)=xe^{-2x}\cos(x)$$
$$e^{-x}(y''-y')-e^{-x}(y'-y)=xe^{-2x}\cos(x)$$
$$(e^{-x}y')'-(e^{-x}y)'=xe^{-2x}\cos(x)$$
$$(e^{-x}y'-e^{-x}y)'=xe^{-2x}\cos(x)$$
$$(e^{-x}y)''=xe^{-2x}\cos(x)$$
A: I would try a particular solution of the form
$$
y_p(x)=e^{-x}[(ax+b)\cos x+(cx+d)\sin x]
$$
A: Variation of Parameters & the Wronskian
We know the general solution of the complementary $\boxed{y_0(x) = c_1e^x+c_2xe^x } \quad(1)$
Theorem

Let the ODE $a_ny^{(n)} +a_{n-1}y^{(n-1)} + \cdots + a_1y' +a_0y =b$ and let $\{y_1,y_2\}$ be a fundamental set of solutions of the ODE. Then a partial solution of the ODE is:
$$ y_p(x)=\sum_{i=1}^{2}y_i(x)\int_{x_0}^{x} \frac{W_i(y_1,y_2)(t)}{W(y_1,y_2)(t)}\cdot \frac{b(t)}{a_n(t)} dt$$ where $W$ i the Wronskian determinant

Choose $x_0 =0$.

*

*$ W(y_1,y_2)(t)  = 
\left| \begin{array}{ccc}
y_1 & y_2  \\
y^{'}_1  &  y^{'}_2   
 \end{array} \right| =
\left| \begin{array}{ccc}
e^t & te^t \\
e^t &  e^t(1+t)
 \end{array} \right| = e^{2t}(1+t) - e^{2t}t = e^{2t}
$


*$ W_1(y_1,y_2)  = 
\left| \begin{array}{ccc}
0 & te^t \\
1 &  e^t(1+t)
 \end{array} \right| = -te^t
$


*$ W_2(y_1,y_2)  = \left| \begin{array}{ccc}
e^t & 0 \\
e^t &  1
 \end{array} \right| = e^t$
Furthermore $a_n(t) = 1$ and $b(t) = te^{-t}cos(t)$
Hence the partial solution is:
$$ y_p(x)=\sum_{i=1}^{2}y_i(x)\int_{0}^{x} \frac{W_i(y_1,y_2)(t)}{W(y_1,y_2)(t)}\cdot \frac{b(t)}{a_n(t)} = \\
 e^x \int_{0}^{x} \frac{-te^t}{e^{2t}}\cdot te^{-t}cos(t) dt + xe^x \int_{0}^{x} \frac{e^t}{e^{2t}}\cdot te^{-t}cos(t) dt  (1) $$
Solving those on Wolfram Alpha I1, I2:
$y_p(x) = \frac{e^{-x}}{125} (- 2 (10 x + 11) sin(x) + (15 x + 4) cos(x))$

Note: This is the same result as the @Aryadeva's result who used another method on her/his answer.
