How find this ODE:$y''-y'+y=x^2e^x\cos{x}$ Find this follow ODe solution
$$y''-y'+y=x^2e^x\cos{x}$$
I konw  solve this follow three case
$$y''-y'+y=x^2\cos{x}$$
$$y''-y'+y=e^x\cos{x}$$
$$y''-y'+y=x^2e^x$$
But for $f(x)=x^2e^x\cos{x}$, I can't.
Thank you  very much!
 A: Honestly, I don't know how the three cases you made is related to the original ODE. I point you a criteria:

Let $a_ny^{(n)}+a_{n-1}y^{(n-1)}+\cdots+a_1y'+a_0y=Q(x)$ where $a_n\neq0$ and $Q(x)$ is not zero in an interval $I$.  If No term of $Q(x)$ is the same as a term of $y_c(x)$, then the particular solution $y_p(x)$ is a linear combination of the terms in $Q(x)$ and all its linearly independent derivatives.

Here, we have $y_c(x)=\exp(x/2)\left(C_1\sin(\frac{\sqrt{3}}{2})+C_2\cos(\frac{\sqrt{3}}{2})\right)$ so according to what above point says we get: $$y_p(x)=(Ae^x+Bxe^x+Cx^2e^x)\sin(x)+(De^x+Exe^x+Fx^2e^x)\cos(x)$$ Now we can use the Undetermined coefficient method to find $A,B,C,D,E,F$. Let's to find $y_p'(x)$ and $y_p''(x)$:
$$y_p(x)=(A\exp(x)+B\exp(x)+Bx\exp(x)+2Cx\exp(x)+Cx^2\exp(x))\sin(x)+(A\exp(x)+Bx\exp(x)+Cx^2\exp(x))\cos(x)+(D\exp(x)+E\exp(x)+Ex\exp(x)+2Fx\exp(x)+Fx^2\exp(x))*cos(x)-(D\exp(x)+Ex\exp(x)+Fx^2\exp(x))\sin(x)$$ and $$y''_p(x)=(A\exp(x)+2B\exp(x)+Bx\exp(x)+2C\exp(x)+4Cx\exp(x)+Cx^2\exp(x))\sin(x)+(2(A\exp(x)+B\exp(x)+Bx\exp(x)+2Cx\exp(x)+Cx^2\exp(x)))\cos(x)-(A\exp(x)+Bx\exp(x)+Cx^2\exp(x))\sin(x)+(D\exp(x)+2E\exp(x)+Ex\exp(x)+2F\exp(x)+4Fx\exp(x)+Fx^2\exp(x))\cos(x)-(2(D\exp(x)+E\exp(x)+Ex\exp(x)+2Fx\exp(x)+Fx^2\exp(x)))\sin(x)-(D\exp(x)+Ex\exp(x)+Fx^2\exp(x))\cos(x)$$ Now we have $$y''_p(x)-y'_p(x)+y_p(x)=\exp(x)(2\sin(x)Cx+\cos(x)Bx+4\cos(x)Cx+\cos(x)Cx^2+2\cos(x)Fx-\sin(x)Ex-4\sin(x)Fx-\sin(x)Fx^2+\sin(x)B+2\sin(x)C+\cos(x)A+2\cos(x)B+\cos(x)E+2\cos(x)F-\sin(x)D-2\sin(x)E)$$ If we make the latter identity equal to $x^2\exp(x)\cos(x)$ we get $$2C-E-4F=0,~~B+4C+2F=0,~~C=1,~~-F=0,~~B+2C-D-2E=0,~~A+2B+E+2F=0$$ and finally, $$A=6,B=-4,C=1,F=0,E=2,D=-6$$
A: $\newcommand{\+}{^{\dagger}}%
 \newcommand{\angles}[1]{\left\langle #1 \right\rangle}%
 \newcommand{\braces}[1]{\left\lbrace #1 \right\rbrace}%
 \newcommand{\bracks}[1]{\left\lbrack #1 \right\rbrack}%
 \newcommand{\ceil}[1]{\,\left\lceil #1 \right\rceil\,}%
 \newcommand{\dd}{{\rm d}}%
 \newcommand{\ds}[1]{\displaystyle{#1}}%
 \newcommand{\equalby}[1]{{#1 \atop {= \atop \vphantom{\huge A}}}}%
 \newcommand{\expo}[1]{\,{\rm e}^{#1}\,}%
 \newcommand{\fermi}{\,{\rm f}}%
 \newcommand{\floor}[1]{\,\left\lfloor #1 \right\rfloor\,}%
 \newcommand{\half}{{1 \over 2}}%
 \newcommand{\ic}{{\rm i}}%
 \newcommand{\iff}{\Longleftrightarrow}
 \newcommand{\imp}{\Longrightarrow}%
 \newcommand{\isdiv}{\,\left.\right\vert\,}%
 \newcommand{\ket}[1]{\left\vert #1\right\rangle}%
 \newcommand{\ol}[1]{\overline{#1}}%
 \newcommand{\pars}[1]{\left( #1 \right)}%
 \newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}}
 \newcommand{\pp}{{\cal P}}%
 \newcommand{\root}[2][]{\,\sqrt[#1]{\,#2\,}\,}%
 \newcommand{\sech}{\,{\rm sech}}%
 \newcommand{\sgn}{\,{\rm sgn}}%
 \newcommand{\totald}[3][]{\frac{{\rm d}^{#1} #2}{{\rm d} #3^{#1}}}
 \newcommand{\ul}[1]{\underline{#1}}%
 \newcommand{\verts}[1]{\left\vert\, #1 \,\right\vert}$
In general, we want to solve
$\ds{{\rm y}''\pars{x} - {\rm y}'\pars{x} + {\rm y}\pars{x} = \fermi\pars{x}}$ where $\fermi$ is a given function. Let's assume, for simplicity, we know the 'boundary conditions'
$$
{\rm y_{p}}\pars{0}\quad\mbox{and}\quad{\rm y_{p}}'\pars{0}\,,\qquad
\mbox{( other cases can be handled in a similar way )}\tag{1}
$$
The solution is given by:
$$
{\rm y}\pars{x}
={\rm y_{p}}\pars{x} + \int_{-\infty}^{\infty}{\rm G}\pars{x,t}\fermi\pars{t}\,\dd t
\tag{2}
$$
where ${\rm y_{p}}\pars{x}$ satisfies
$\ds{{\rm y_{p}}''\pars{x} - {\rm y_{p}}'\pars{x} + {\rm y_{p}}\pars{x} = 0}$ and the 'boundary conditions' $\pars{1}$. ${\rm G}\pars{x,t}$ satisfies:
$$
\pars{\totald[2]{}{x} - \totald{}{x} + 1}{\rm G}\pars{x,t} = \delta\pars{x - t}\,,
\qquad
\left\vert%
\begin{array}{rcl}
{\rm G}\pars{0,t} & = & 0
\\[2mm]
\left.\partiald{{\rm G}\pars{x,t}}{x}\right\vert_{x\ =\ 0} & = & 0
\end{array}\right.\tag{3}
$$
The $\ds{{\rm G}\pars{x,t}}$ differential equation in $\pars{2}$ is equivalent to:
$$\left\lbrace%
\begin{array}{rcl}
\pars{\totald[2]{}{x} - \totald{}{x} + 1}{\rm G}\pars{x,t} = 0
& \mbox{if} & x \not= t
\\[2mm]
\left.\partiald{{\rm G}\pars{x,t}}{x}\right\vert_{x\ =\ t^{-}}^{x\ =\ t^{+}}
= 1&&
\end{array}\right.\tag{4}
$$
Solutions of $\ds{{\rm y}''\pars{x} - {\rm y}'\pars{x} + {\rm y}\pars{x} = 0}$ has the form $\ds{\expo{x/2}\bracks{A\sin\pars{kx} + B\cos\pars{kx}}}$ with
$\ds{k = \root{3}/2}$. Then
$$
{\rm G}\pars{x,t}
=
\left\lbrace%
\begin{array}{lcl}
\expo{x/2}\bracks{A\sin\pars{kx} + B\cos\pars{kx}} & \mbox{if} & x < t
\\
\expo{x/2}\bracks{C\sin\pars{kx} + D\cos\pars{kx}} & \mbox{if} & x > t
\end{array}\right.
$$
The ${\rm G}\pars{x,t}$ 'boundary conditions' in $\pars{3}$ leads to $A = B = 0$.
${\rm G}\pars{t^{-},t} = {\rm G}\pars{t^{+},t}$ and the 'boundary condition' in $\pars{4}$ lead to:
\begin{align}
&\expo{t/2}\bracks{C\sin\pars{kt} + D\cos\pars{kt}} = 0 
\\[3mm]&
\half\,\expo{t/2}\bracks{C\sin\pars{kt} + D\cos\pars{kt}}
+
k\expo{t/2}\bracks{C\cos\pars{kt} - D\sin\pars{kt}} = 1
\end{align}
which are equivalent to:
$$
\left.
\begin{array}{rcrcl}
\sin\pars{kt}C & + & \cos\pars{kt}D & = & 0
\\
\cos\pars{kt}C & - & \sin\pars{kt}D & = & {\expo{-t/2} \over k}
\end{array}\right\rbrace\
\imp\
\left\lbrace%
\begin{array}{rcl}
C & = & \phantom{-}{\expo{-t/2}\cos\pars{kt} \over k}
\\[2mm]
D & = & -\,{\expo{-t/2}\sin\pars{kt} \over k}
\end{array}\right.
$$
and
$$
{\rm G}\pars{x,t}
=\left\lbrace%
\begin{array}{lcl}
0 & \mbox{if} & x < t
\\[2mm]
{\expo{\pars{x - t}/2}\sin\pars{k\bracks{x - t}} \over k} & \mbox{if} & x > t
\end{array}\right.
$$
$\ds{{\rm y}\pars{x}
=
{\rm y_{p}}\pars{x} + {\expo{x/2} \over k}
\int_{-\infty}^{x}\expo{-t/2}\sin\pars{k\bracks{x - t}}\fermi\pars{t}\,\dd t
=
{\rm y_{p}}\pars{x} - 
{1 \over k}\int_{-\infty}^{0}\expo{-t/2}\sin\pars{kt}\fermi\pars{t + x}\,\dd t}$
$$\color{#0000ff}{\large%
{\rm y}\pars{x}
=
{\rm y_{p}}\pars{x} + {1 \over k}
\int_{0}^{\infty}\expo{t/2}\sin\pars{kt}\fermi\pars{x - t}\,\dd t\,,
\qquad k = {\root{3} \over 2}}
$$
$\tt\mbox{Just plug your function}\ \fermi\pars{x - t}\ \mbox{in this expression !!!}$
A: One more way to solve it is using the Laplace transformation $\mathcal{L}.$
Let $\mathcal{L}(y)=Y(p).$  Then
$$
\mathcal{L}(y')=p Y(p)-y(0),\\
\mathcal{L}(y'')=p^2 Y(p)-y'(0)-p y(0),\\ \text{and, using the standard rules  for L.t.}\\
\mathcal{L}({x}^{2}{{\rm e}^{x}}\cos \left( x \right))=\frac{1}{\left( p-1-i \right) ^{3}}+ \frac{1}{\left( p-1+i \right) ^{3}}
$$
We have  now 
$$
 \left( {p}^{2}-p+1 \right) Y \left( p \right) -py \left( 0 \right) +y
 \left( 0 \right) -  y'  \left( 0 \right) =
 \frac{1}{\left( p-1-i \right) ^{3}}+ \frac{1}{\left( p-1+i \right) ^{3}},
$$
or 
$$
Y(p)={\frac {1}{ \left( p-1-i \right) ^{3} \left( {p}^{2}-p+1 \right) }}+{\frac {1}{ \left( p-1+i \right) ^{3} \left( {p}^{2}
-p+1 \right) }}-{\frac {-py \left( 0 \right) +y \left( 0 \right) -
 y''   \left( 0 \right) }{{p}^{2}-p+1}}.
$$
To find $y$ we  compute the inverse Laplace transform for $Y(p).$ After some calculation we get
$$
y(x)=\mathcal{L}^{-1}(Y(p))=\left( 2\,\cos \left( x \right)  \left( -3+x \right) +\sin \left( x \right)  \left( 6-4\,x+{x}^{2} \right)  \right) {{\rm e}^{x}}+1/3\,
 \left( 3\,\cos \frac{\sqrt {3}x}{2}  \left( 6+y \left( 0
 \right)  \right) +\sin \frac{\sqrt {3}x}{2}  
 \left( -10+2\, y''   \left( 0 \right) -y \left( 0
 \right)  \right)  \right) {{\rm e}^{\frac 12\,x}}.
$$
After simplification taking into account that $y(0), y'(0)$  are constant we get 
$$
y(x)={{e}^{\frac 12\,x}}(\sin \frac{\sqrt {3}x}{2} { C_1}+\cos \frac{\sqrt {3}x}{2} {C_2})+ \left( 2\,\cos \left( x \right)  \left( x-3
 \right) +\sin \left( x \right)  \left( 6-4\,x+{x}^{2} \right) 
 \right) {{ e}^{x}},
$$
for  some constant $C_1, C_2.$
