Problem from Zorich's book Choose real numbers $a$ and $b$ so that the function $f(x)=\cos x - \dfrac{1+ax^2}{1+bx^2}$ is an infinitesimal of highest possible order as $x\to 0$.
My solution: Actually this problem is not so difficult and I guess that I've almost solved it. We'll split the solution into some cases:

*

*If $b=0$ and using Taylor's series one can write $f(x)=x^2(-1/2-a)+o(x^3)$ as $x\to 0$.

So we see that if $a=-1/2$ then $f(x)=o(x^3)$. If $a\neq -1/2$ then $f(x)=o(x)$ as $x\to 0$.


*Suppose that $b\neq 0$ then again using Taylor's series one can derive that $$f(x)=\cos x-(1+ax^2)(1+bx^2)^{-1}=x^2\left(-\frac{1}{2}-a+b\right)+x^4\left(\frac{1}{24}-b^2+ab\right)+x^6\left(-\frac{1}{720}-ab^2+b^3\right)+o(x^7).$$
One can show that if $a=-\dfrac{5}{12}$ and $b=\dfrac{1}{12}$ then $f(x)=o(x^5)$ as $x\to 0$.
Definition: If $f=o(g)$ and g is itself infinitesimal as $x\to 0$, we say that $f$ is an infinitesimal of higher order than $g$ as $x\to 0$.
Question: Can anyone explain in details which one has the highest possible order? I guess it should be in the case when $a=-\dfrac{5}{12}$ and $b=\dfrac{1}{12}$ but cannot explain it in a rigorous way. If someone can explain it in a great detail that would be awesome!
 A: Regardless of whether $b = 0$ or not, we can always write $$f(x) = \cos(x)  - (1 + ax^2)(1 - bx^2 + b^2 x^4 - b^3 x^6 + \dots)$$ which leads to $$f(x) = (-a + b - 1/2)x^2 + (ab - b^2 + 1/24)x^4 + (-ab^2 + b^3 - 1/720)x^6 + \dots$$
Therefore for $a = -5/12$ and $b = 1/12$, we have $f(x) = 1/480 x^6 + \dots$ which has order of vanishing $6$.
On the other hand, if $f$ has order $\geq 6$, then we must have $−a+b−1/2 = 0$ and $ab−b^2+1/24 = 0$. It is easy to solve from these two equations that $a = -5/12$ and $b = 1/12$ is the only possibility.
A: You've chosen the unique $a$ and $b$ to make the coefficients of $x^2$ and $x^4$ vanish (I presume — I haven't checked) in the Taylor expansion. If you make any other choices, you'll have $f(x) = O(x^2)$ or $f(x)=O(x^4)$ instead. Getting $f(x)=O(x^6)$ is the optimal result.
A: If $f\in o(x^5)$, $x \to 0$, then $f(x)=\varepsilon(x) x^5 $, where $\lim\limits_{x\to x_0} \varepsilon(x) = 0$. Hence we can write representation $f(x)=\varepsilon(x) x^2 \cdot x^3= \varepsilon_1(x)\cdot x^3$, where  $\lim\limits_{x\to x_0} \varepsilon_1(x) = 0$, so we can say, that $f\in o(x^3)$ i.e. $o(x^5) \subset o(x^3)$. So, being in $o(x^5)$ means to be more higher order, then being in $o(x^3)$.
