Find parameters $a,b$ such that $x^6-2 x^5+2 x^4+2 x^3-x^2-2 x+1-\left(x^3-x^2+a x+b\right)^2>0$ The probrem is to prove that
$$x^6-2 x^5+2 x^4+2 x^3-x^2-2 x+1>0.$$
(the minimum value is about 0.02, tested by wolframalpha.)
I use sos(sum of squares) method, my idea is to reduce the degree of the polynomial gradually.
First I need to find $a,b$ so that
$$x^6-2 x^5+2 x^4+2 x^3-x^2-2 x+1-\left(x^3-x^2+a x+b\right)^2>0.$$
It's equivalent to
$$(1-2 a) x^4+(2 a-2 b+2)x^3 +\left(-a^2+2 b-1\right)x^2 +(-2 a b-2)x -b^2+1>0,$$ or
$$
\left( (1-2\!\:a)\!\:x^4+(2+2\!\:a-2\!\:b)\!\:x^3+(-1-a^2+2\!\:b)\!\:x^2+(-2-2\!\:a\!\:b)\!\:x+1-b^2 \right) _{\min}>0.$$
Let
$$f(x)=
(1-2\!\:a)\!\:x^4+(2+2\!\:a-2\!\:b)\!\:x^3+(-1-a^2+2\!\:b)\!\:x^2+(-2-2\!\:a\!\:b)\!\:x+1-b^2$$
and $f'(x)=0$, I get
$$
-2-2\!\:a\!\:b+2\!\:(-1-a^2+2\!\:b)\!\:x+3\!\:(2+2\!\:a-2\!\:b)\!\:x^2+4\!\:(1-2\!\:a)\!\:x^3=0.
$$ However, it's difficult  and ugly to solve the equation. And it's impossible to solve the extreme point when the degree of the polynomial increases, for example, prove that $$x^{12}-x^{11}+x^8-x^7+x^6+x^3-2 x+1>0,$$ we need to find 5 parameters $a,b,c,d,e$ such that $$
x^{12}-x^{11}+x^8-x^7+x^6+x^3-2x+1-\left( x^6-\frac{x^5}{2}+ax^4+bx^3+cx^2+dx+e \right) ^2>0.
$$
How can I solve the problem? Is there any general method?
 A: For the original problem, observe that
$$x^6 - 2x^5 + 2x^4 + 2x^3 - x^2 - 2x + 1 = (x^3 - x^2)^ 2 + (x^2 +x - 1)^2.$$
Since both squares cannot be 0 at the same time, hence it is strictly positive.

For your stated problem, $ a, b = 0 $ is still not a solution to your strict inequality because the expression does hit 0 when $x^2 + x - 1 = 0 $.
Testing values near that point indicates that $ a = -0.01, b = 0.01$ works.
The resulting quartic has 4 complex roots, hence is strictly positive.
Proof by Wolfram Alpha since it's not that interesting.
Given the above, I'm not motivated to figure out the rest of the SOS.

Notes

*

*You do not always have to "reduce the degree of the polynomial gradually".

*

*EG We could have (say) $ f(x) = (0.6x^3 + g(x) ) ^2 + (0.8 x^3 + h(x) ) ^2 $ for some quadratics $g(x), h(x)$.

*EG Had you tried to do so, you'd likely have ended up with $ (x^3 - x^2 + 0.5 x + 1.5)^2$ in order to remove the $x^4$ and then $x^3 $ term. However, this leaves us with $1.75x^2 - 3.5x - 1.25$ which has distinct real roots, so we can't SOS further.



*I chanced upon this identity, in part from recognizing some of the terms. You guessed that $(x^3-x^2)^2$ might be involved, so looking at $ f(x) - (x^3 -x^2)^2$  would have been a good next step.

*The minimum of $f(x)$ occurs near the positive root of $x^2 + x - 1 = 0$. This shouldn't be too surprising based on the identity.

A: Let $p(x)=x^6-2x^5+2x^4+2x^3-x^2-2x+1$. A possible way to show $p(x)>0$ for all $x\in\mathbb{R}$ follows. Consider 3 cases.
Case 1: $x<-1$. Let $r(x)=p(x-1)$. Then $r(x)=x^6-8x^5+27x^4-46x^3+40x^2-18x+5$. Clearly, $r(x)>0$ for $x<0$, so $p(x)=r(x+1)>0$ for $x<-1$.
Case 2: $x>\frac 12$. Let $q(x)=p\left(x+\frac 12\right)$. Then $q(x)=x^6+x^5+\frac34x^4+\frac72x^3+\frac{55}{16}x^2-\frac{15}{16}x+\frac{5}{64}$. We have $\frac{55}{16}x^2-\frac{15}{16}x+\frac{5}{64}>0$ (check by completing the square or computing the discriminant) so $q(x)>0$ for $x>0$ which implies that $p(x)=q\left(x-\frac 12\right)>0$ for $x>\frac 12$.
Case 3: $-1\le x\le \frac 12$. Write $p(x)=x^4(x^2-2x+2)+(2x-1)(x^2-1)$. We have $x^4(x^2-2x+2)=x^4((x-1)^2+1)\ge 0$ with equality only if $x=0$ and $(2x-1)(x^2-1)\ge 0$ for $-1\le x\le \frac 12$ with equality only if $x=\frac 12$ or $x=-1$, so $p(x)>0$ in this case as well.
