$$ \int_a^b f(x) \,\mathrm dx $$ now make the $u$-substitution $$u \mapsto c + (x-a)(x-b).$$
Typo: judging from your post, your substitution is actually \begin{align}u=g(x)&=c + (x-a)(x-b)\tag# \\ g'(x)&=2x-a-b\\g(a)&=c=g(b).\end{align}
The resulting integral is $$ \int_c^c h(u) \,\mathrm du =0.$$
what's wrong with this substitution such that, regardless of the starting integrand $f,$ it results in $0\ ?$
It is indeed correct that $$ \int_a^b f(x) \,dx\overset{u = c + (x-a)(x-b)}{=}0.\tag✓$$ However, this is not “regardless of $f,$” but whenever the substitution is even possible, which is precisely when $f$ has rotational symmetry about $\left(\frac{a+b}2,0\right)$ and $h$ is integrable, and has an antiderivative, on $g[a,b].$ (The stipulation on $h$ means that the red bit in $$ \int_c^c \color{red}{\text{[this integrand doesn't matter]}} \,dx=0$$ is false.) So, if $f$ lacks the stipulated symmetry property, then there is no basis for, and no way to, make the given substitution $(\#).$
Proof
Let's begin with this strong version of the integration-by-substitution theorem, which does not generally require injectivity:
- If $g'$ is integrable on $[a,b]$ and $h$ is integrable, and has an antiderivative, on $g[a,b],$ then $$\int_a^bh\big(\color{violet}{g(x)}\big)\,\color{cyan}{g'(x)}\,\mathrm{d}x=\int_{g(a)}^{g(b)}h(u)\,\mathrm{d}u.$$
This corresponds exactly to your above work, with $$f(x)=h\Big(\color{violet}{c + (x-a)(x-b)}\Big)\,\color{cyan}{(2x-a-b)}.$$ As $f$ has not been concretely given, function $h$ is necessarily abstract (though we note that the choice of $h$ is also to satisfy the theorem's conditions; this immediately happens if $h$ is continuous on $g[a,b].)$ Now let's shift $f$ by $\displaystyle\frac{a+b}2$ units leftwards: $$f_N(x)=f\left(x+\frac{a+b}2\right)\\
=h\Bigg(c + \left(\left(x+\frac{a+b}2\right)-a\right)\left(\left(x+\frac{a+b}2\right)-b\right)\Bigg)\:\:\left(2\left(x+\frac{a+b}2\right)-a-b\right)\\
=h\left(c+x^2-\frac{(a-b)^2}4\right)\;\;2x.$$ Since $f_N(-x)=-f_N(x),$ the new function is odd. Thus, $f$ has rotational symmetry about the point $\left(\frac{a+b}2,0\right).$
Addendum
In the youtube video Weird Integral Trick, @blackpenredpen demonstrates that $$u=g(x)=x(x-a-b)$$ is a wonderful substitution, because it amazingly and universally zeroes out every integral: $$\int_a^b f(x)\,\mathrm dx\overset{u=x(x-a-b)}{=}\int_{-ab}^{-ab}\text{[this integrand doesn't matter]}\,\mathrm du=0.$$
The apparent anomaly is currently explained in neither the video nor its comments, so, for the record, it has the same explanation as above: this substitution is possible precisely when both these conditions are satisfied: 1) $f$ on $[a,b]$ has rotational symmetry about $\left(\frac{a+b}2,0\right),$ 2) $h$ on $g[a,b]$ is integrable and has an antiderivative; if so, then the result $0$ is correct.