What is the significance of unstable zeros in control system? I am new to Systems Theory, I recently learnt about inertia of a polynomial which gives the number of stable, antistable and imaginary zeros. I am just trying to understand if there is a point in knowing how many antistable zeros are there? I guess what I am trying to ask is, can we do anything about the antistable zeros?
Just trying to understand the practical application of this.
Thanks!
 A: The best ways of thinking about the effect of zeros are the Root Locus and the Bode diagram.
The zeros are the end point of the root locus. Thus a system with zeros whose real parts are not negative will become unstable if a feedback loop with sufficiently high gain is closed. Such zeros impose limits on the performance of a stabilizing controller.
Compared to a minimum-phase system whose Bode magnitude plot is identical, a nonminimum-phase system, that is, one with unstable zeros, will have extra phase of up to 180 degrees for each unstable zero. The extra phase corresponds to extra turns around the origin in the Nyquist diagram. That also explains how right half plane zeros make a system harder to control by feedback.
Additionally, zeros cannot be be moved by feedback, the way poles can be assigned. Zeros with nonnegative real part cannot be canceled, because the cancellation would be unstable. Thus the designed has to live with them, as a fundamental limitation on our ability to design a feedback controller.
(I should add that time-domain linear quadratic control is of limited use in understanding the role of zeros.)
A: Let's say your systems is linear time invariant (LTI).
It's transfer function is 
$${Y(s) \over X(s)}=k{(s-z_1)\dots(s-z_m) \over (s-p_1)\dots(s-p_n)}, \quad n\ge m$$
What cause instability are poles $p_i$ with none negative real parts. The reason is that in the impulse response, you would see terms such as $e^{p_it}$. These terms are not damped for $\Re\{p_i\}\ge 0$.
The numerator zeroes with positive real part are undesirable too (non-minimum phase systems). But the stability is determined by the denominator not numerator.
What is the solution?
If your plant is G and your feedback is H, the closed loop is
$${Y(s) \over X(s)}={G \over 1+GH}$$
What determines stability are poles of ${G \over 1+GH}$ or in another word, the roots of $1+GH=0$.
You can add a compensator $C$ before $G$ to change the roots to $(1+CGH=0)$.
You might change the gain of feedback by replacing $H$ with $kH$ then choose $k$ where roots of $(1+kGH=0)$ are stable.
Another method is using Linear Quadratic Regulator (LQR) to perform optimal feedback control. I refer you to [1], [2], [3].

