What is the physical meaning of Bode plot in case of unstable system? I know that from the mathematical point of view it doesn't matter if we plot Bode diagram of stable or unstable system. It's just a function of complex value.
However from the physical point of view, Bode plot shows steady response to a sinusoidal input. If the system is unstable there is no steady response, am I right? So is it only a "magic" of mathematics that although there is no meaning, it's still useful to use it for design?
 A: The 'frequency response' interpretation of a Bode Plot only holds for stable (or marginally stable) systems. If you choose to draw it for an unstable system, then it does not have the usual physical interpretation. Specifically, it is NOT a frequency response (the steady-state amplitude and phase response to a sinusoidal input).
There is a simple mathematical reason why. The Bode plot at a particular frequency, $\omega$, is defined as being the system transfer function $G(s)$, evaluated at $s = j\omega$. However, for an unstable system, this evaluation violates the assumptions under which the original transfer function was derived using the Laplace Transform. Remember that Laplace Transforms always have allowable values of $s$, called the "Region of Convergence". The Region of Convergence for the Laplace Transform of an unstable system is $Re(s) \geq a$, where $a>0$. Therefore, when you use the $s = j\omega$ substitution, you are violating the Region of Convergence condition. What you get is no longer the Fourier Transform of a time-domain signal and should not be interpreted as such.
Certain control analysis techniques use the $s = j\omega$ substitution for unstable systems, but it is just mathematical trickery and the result is never interpreted as a Fourier Transform. The Nyquist Criterion is an example: this is a mathematical 'trick' that just helps visualise the stability of a closed-loop system. The Nyquist Criterion can be checked using a Bode Plot, which is where you may come across the Bode Plot of an unstable system. But when handling such a Bode Plot, always bear in mind that it is NOT a frequency response.     
A: You can interpret that Bode plot as the response to a sinusoidal input even for unstable systems, if instead of "steady state" you say "with initial conditions exactly zero".
For a stable system, those two interpretations coincide. Remember that the space of all solutions of an inhomogenous system of linear differential equations $f'(t) = Af(t) + g(t)$ can always be written as $$
  \lambda_1f_1(t) + \ldots + \lambda_nf_n(t) + f_p(t)
$$
where $f_1,\ldots,f_n$ are (linearly independent) solutions of the homogenous system $f'(t) = Af(t)$, and $f_p$ is one particular solution of the inhomogenous system. Also remember that $\lambda_1,\ldots,\lambda_n$ are determined by the initial condition of the system. Now, if the system is stable, all the $f_1,\ldots,f_n$ decay exponentially. Thus, no matter what the initial conditions are, if $t$ gets large enough, the solution will basically just be $f_p(t)$.
For an unstable system, some of the $f_1,\ldots,f_n$ grow exponentially (or are constant), so you cannot count on their influence to vanish at some point. You can, however, simply decide to set $\lambda_1=\ldots=\lambda_n=0$ in the solution, and hence look only at $f_p$.
Take, for example, the simple system $$
f'(t) = f(t) + e^{i\omega t}
$$
The corresponding homogenous system has the solution $f_1(t)=e^t$ and is thus unstable. A particular solution of the inhomogenous system is $$
  f_p(t)=-\frac{1+i\omega}{1+\omega^2}e^{i\omega t}
$$
and the frequency response (i.e. what you plot in a Bode plot) is therefore $$
  A(\omega) = -\frac{1+i\omega}{1+\omega^2}
$$
Note how it is perfectly reasonable to interpret this as the attenuation of the sinusoidal input $e^{i\omega t}$ - it is, after all, derived from an actual solution of the differential equation. You just won't be able to measure it, because in every actual experiment the initial condition (i.e. $\lambda_1$) will never be exactly zero - and once it differs from zero only the slightest, $\lambda_1f(t)=\lambda_1e^t$ will quickly dominate the result.
A: The bode plot for an unstable system does have a physical interpretation. If we design a controller such that the closed loop system is stable then we can infer something important from the bode plot of the unstable system. The bode plot of the unstable plant, in such a case, would give the steady state response of the plant for sinusoidal system inputs of different frequencies, assuming closing the loop stabilizes the system. 
You can try simulating this in simulink for a simple system. Let the Plant, P = 1/s-1, K = 2 and close the loop. Generate  a reference signal such that the input to P is a unit step (approximately). A step input (at the reference) of magnitude -0.5 should work, if you connected K and P in series. Then, the steady state plant output would be -1. Hence, though the system is unstable, it does make sense to talk about it's dc gain and bode plot.
A previous answer regarding particular solution, as pointed out by jkn, is inaccurate. The argument would hold, I think, only when the particular solution(or the 'forced response') and the zero state response of the system are the same, which is generally not true.
A: I think both fgp's and Swarmy's answers are reasonable. My conclusion on the matter is, bode plot for an unstable system shows exactly the same thing as that for a stable system, which is the input-output relationship for a sinusoidal input under certain frequency.
The ambiguity here might come from that, for a stable LTI system and a sinusoidal input u(t), using bode diagram we can define an plant solution x*(t), aka the steady-state response, which is stable in the sense that, no matter what the initial state of the system x(0) is, the system output x(t) will always converge to the steady-state response, i.e. x(t)->x*(t) as t->inf.
However, one thing we should note is that there exists a initial condition of the plant such that the plant solution is exactly the solution x*(t), where we only need to set x(0)=x*(0). And this applies to unstable system as well. The only problem of unstable system is, if you have any error in the initial condition, the effect of it will increase over time and the solution x(t) diverges from x*(t). It is simply impossible to realise that relationship through experiment or even simulation.
To further demonstrate my idea, consider the example given by Swarmy. Its state-space realisation is xdot=x+u. Try substituting u(t)=sin(t) and x(t)=1/sqrt(2)*sin(t-3/4*pi) and find out whether it satisfies the differential equation. If it does, it means the output is indeed a solution of the system excited by input u(t). Furthermore, the input and output are indeed sinusoids. And you can check the bode diagram of that system indeed indicate the gain and phase changes here. Therefore, bode diagram for unstable systems indeed has a clear physical meaning: it shows the input-output relationship of the plant, just like that for stable systems.
