Definition of inner product of two polynomials I'm reading a paper related to optimization and polynomials. I faced a new think that is not defined in the paper, so I think it should be famous.
What does it mean $\left <p,q\right >\geq0$ when $p$ and $q$ are polynomials?
 A: The usual inner product on polynomials is the $L^2$ inner product.  Eliding quite a bit of detail, if $X$ is some space, then $L^2(X)$ is the set of functions on $X$ that have finite square integrals, i.e.
$$ L^2(X) = \left\{ f : X \to \mathbb{C} \ \middle|\ \int_X |f|^2 < \infty \right\}. $$
(The same definition works with $f : X \to \mathbb{R}$.)  The space of $L^2$ functions is a vector space with a norm given by the integral above.  Moreover, it turns out that we can define an inner product by setting
$$ \langle f,g \rangle = \int_{X} f \overline{g}, $$
where $\overline{g}$ represents the complex conjugate.
To answer your question, first note that if $p$ is a polynomial, then
$$ \int_{\mathbb{C}} |p|^2 = \infty, $$
which means that polynomials are not $L^2$ functions on $\mathbb{C}$.  On the other hand, if $X \subseteq \mathbb{C}$ is any compact set—in particular, if $X = [a,b] \subseteq \mathbb{R}$—then polynomials will have bounded $L^2$ norm on $X$, i.e.
$$ \int_{a}^{b} |f(x)|^2\,\mathrm{d}x < \infty. $$
This can be seen by noting that polynomials are continuous, and therefore bounded on compact sets.  Hence the integral above is at most the square of the maximum of the polynomial times the length of the integral.
Therefore the most likely explanation is that $p,q\in L^2([a,b])$ (or, perhaps more generally, $p,q\in L^2(X)$ where $X$ is some compact connected set in $\mathbb{C}$), and the inner product is given by
$$ \langle p,q\rangle = \int_{a}^{b} p(x) q(x) \,\mathrm{d}x. $$
Note that we have dropped the conjugation, since $q(x)\in \mathbb{R}$ implies that $\overline{q(x)} = q(x)$.
