$y'''-y=x^{2}$ has solution -- `"multiplicity"`? The page 667 of the book (sorry not in English) claims $y'''-y=x^{2}$ to have the solution
$$y(x)=C_{1}e^{x}+e^{-x/2}\left(C_{2} \cos \left( \frac{\sqrt{3}x}{2} \right)+C_{3} \sin\left(\frac{\sqrt{3} x}{2}\right)\right) -x^{2}.$$
The book mentions that with the $m$ -multiple real root solution is $x^{k}e^{rx}$ and with $m$ -multiple conjugate pair $\alpha\pm i\beta$ the solution is $x^{k}e^{\alpha x}\cos(\beta x), x^{k} e^{\alpha x} \sin(\beta x)$ where $k=0...m-1$. 
Let's check how to use it in this example. We have 3th order DY so $m=3$. But what kind of roots does this have $r^{3}-1=0$? 
 A: $r=1$ is a solution of $r^3-1$.  So 
$$r^3-1=(r-1)q(x).$$ 
for some quadratic expression $q$.
Doing the division $q(x)={r^3-1\over r-1}$ yields $$r^3-1=(r-1)(r^2+r+1).$$ 
(Or, use the difference of cubes formula on $r^3-1^3$). 
The quadratic formula gives the roots of $r^2+r+1$: $r={-1\over 2}\pm{\sqrt 3\,i\over 2}$.
So, the three roots are: $1$, ${-1\over 2}+{\sqrt 3\,i\over 2}$, and
${-1\over 2}-{\sqrt 3\,i\over 2}$.

You do not have $m=3$ here for the formula you gave.  In that formula, $m$ depends on which root you are looking at and refers to the power of the corresponding factor occurring in the characteristic equation.
Here, the characteristic equation is
$$
r^3-1 =(r-1)^\color{red}1 \textstyle\bigl( r- ({-1\over 2}+{\sqrt 3\,i\over 2})\bigr)^\color{green}1\bigl( r- ({-1\over 2}-{\sqrt 3\,i\over 2})\bigr)^\color{green}1.
$$
For the root $r=1$, we have $m=\color {red}1$. (Which gives the term $C_1 e^x$ in the solution. Note, with $m=1$, the solution formula gives only one term: $x^0e^{rx}=e^{rx}$.) 
For the complex conjugate pair root $r={-1\over 2}\pm{\sqrt 3\,i\over 2}$, we have $m=\color {green}1$. (Which gives the term $e^{-x/2}  \bigl(C_2 \cos(\sqrt3x/2)+C_3\sin(\sqrt3x/2)\bigr)  $ in the solution.)

As another example, suppose one had a homogeneous linear differential equation with constant coefficients that had  characteristic equation:
$$
(r-2)^2(r+3)^3(r+1).
$$
Then the roots are 
$\ \ \ \ r=2$ with $m=2$
$\ \ \ \ r=-3$ with $m=3$
$\ \ \ \ r=-1$ with $m=1$
The general solution to the equation would be
$$
(C_1e^{2x}+C_2 xe^{2x})+(
C_3e^{-3x}+C_4 xe^{-3x}+ C_5x^2e^{-3x})+C_6 e^{-x}.
$$



Your formula is imprecisely stated.  Here is a complete version:

First a definition:
The multiplicity of the root $r$ of the polynomial $q(x)$ is the largest integer $m$ such that $(x-r)^m$ is a factor of $q(x)$.
Now the "recipe":
For the homogeneous equation with constant coefficients:
    $$\tag{1}\def\sss{}
 a_{\sss n}  y^{\sss( n )} +a_{\sss n - 1}  y^{\sss( n - 1 )} +\cdots+a_{\sss1}  y' +a_{\sss0}  y = 0 , \quad a_n\ne0,
 $$
the associated characteristic polynomial (c.p., henceforth)  is
    $$\tag{2}\def\sss{}
 a_{\sss n}x^n+a_{\sss n\!-\!1}x^{n-1}   +\cdots+a_{\sss1}x +a_{\sss0} . 
 $$
To find the general solution of   equation $(1)$: You want to first find a set of $n$, independent, solutions to equation $(1)$. Then you form the general solution by writing it as a general linear combination of the $n$, independent, solutions found.  
Towards this end, you may:


*

*Find the roots and their corresponding multiplicities of the c.p. $(2)$. Complex roots will occur as complex conjugate pairs. We will then speak of the "complex conjugate root $a\pm bi$ with multiplicity $k$", whose meaning is, hopefully, evident.

*Note: If $c$ is a real root of $(2)$ with multiplicity $k $, then
$k$-independent solutions  of $(1)$     are     $$  e^{ct},\ xe^{ct},\ x^2
e^{ct},\ \ldots,\ x^{k-1}e^{ct}.    $$Note that for $k=1$, there is only one term: $e^{ct}$. 

*Note: If $a\pm bi$ is a  complex conjugate pair root of $(2)$ with multiplicity $k$, then
$2k$-independent solutions  of $(1)$     are     $$  e^{at}\sin (bt),\  x
    e^{at}\sin (bt),\ \ldots,\ x^{k-1} e^{at}\sin (bt)   $$    $$
e^{at}\cos (bt),\  xe^{at}\cos (bt)\   ,\ \ldots,\  x^{k-1}e^{at}\cos
(bt).   $$Note that for $k=1$, there are only two terms: $e^{at}\sin(bt)$ and $e^{at}\cos(bt)$.

*Write down all solutions given by steps 2. and 3.: For each real root of the c.p., list the solutions given by step 2; and,  for each complex conjugate pair root, list the solutions given by step 3.  This will generate a set of $n$ independent solutions to equation $(1)$. The general solution to $(1)$ is then
$$y_c=c_{\sss 1}y_{\sss1}+c_{\sss2}y_{\sss2}+\cdots+c_{\sss
    n}y_{\sss n}$$  where $y_1$, $y_2$, $\ldots\,$, $y_n$ are the
$n$-solutions found above.

Once you've developed some facility with this, you should be able to just write down the solution by looking at the fully factored c.p.
A: Your current question has a characteristic polynomial of degree 3. This means that there are 3 roots, when you count multiplicities. In general, let $d$ be the degree of a polynomial $f(x)$, let $r_1, r_2, \ldots, r_n$ be the distinct roots of $f(x)$ with respective multiplicities $m_1, m_2, \ldots, m_n$. Then $d = \Sigma m_i$. 
Since we find three roots (1, $(1+i\sqrt{3})/2$, and $(1-i\sqrt{3})/2$), and their multiplicities must add up to 3 (the degree of our characteristic polynomial), the multiplicity of each root must be 1. Thus we do not need more than k=0 for each root.
If you're looking for an example of a differential equation that has $m \gt 1$, you should look at $y'' - 2y' + 1 = 0$. The characteristic polynomial of this equation has the same root repeated twice, $(r-1)(r-1)$, meaning that the root $r=1$ has multiplicity 2. Thus the homogeneous solution is $y=C_1e^x+C_2xe^x$
A: The roots of the characteristic equation $r^3-1=$ are $r=1$ and $r=(-1\pm i\sqrt3)/2$, so the
general solution of the homogeneous DE $y'''-y=0$ is
$$
y=C_1e^x+e^{-x/2}\left(C_2\cos\left(\frac{\sqrt{3}}{2} x\right)+C_3\sin\left(\frac{\sqrt{3}}{2} x\right)\right).
$$
The usual ansatz for finding a single solution of the non-homogeneous DE works, so I don't see what problems remain?
You seem to be confusing the degree of the equation with the multiplicity of a root?
A: EDIT: What follows is relevant to the question as originally posted. The question was edited after I posted this answer, and this answer is no longer relevant to the revised version of the question. 
It's the $m=2$ case in your notation, so you need $x^ke^{\alpha x}\cos(\beta x)$ and $x^ke^{\alpha x}\sin(\beta x)$ for $k=0..1$.  
