Suppose that we are given following diferential equation $x'=\sin(x)$. Using linear stability analysis, we should find stability points of this equation, we know that fixed points occurs when $f(x)=\sin(x) =0$ or $x=k\pi$ where $k$ is integer. Then in mathematics text book it is written that, $f'(x)=\cos(k\pi)=1$ if $k$ is even,and $f'(x)=\cos(k\pi)=-1$ when $k$ is odd, so $x$ is unstable when $k$ is even, and stable if it is odd. My question is: why it is so? $\cos(x)$ is defined between $[$-$1$..$1$ ], please explain this question more deeply.
Given an ODE $x' = f(x)$. A fixed point is a point where $x' = 0$. This requires $f(x) = 0$. So any roots of the function $f(x)$ is a fixed point.
A fixed point is stable if, roughly speaking, if you put in an initial value that is "close" to the fixed point the trajectory of the solution, under the ODE, will always stay "close" to the fixed point.
A fixed point is unstable if, roughly speaking, for any "threshold" of "nearness" to the fixed point, you can find some solution that starts near the fixed point but becomes far from the fixed point in finite time.
To be even more rough, we can say that a fixed point is stable if the equation of motion $x' = f(x)$ forces a particle to move toward the fixed point, if it starts close to the fixed point, and say that a fixed point is unstable if the equation of motion forces a particle to move away from the fixed point.
Now, to apply the above intuition, remember that the derivative $f'$ is the slope of $f$. So at a fixed point where $f(x) = 0$, if $f'(x) > 0$ we have that $f$ is increasing at $x$, or that $f(x+\epsilon) > 0 > f(x-\epsilon)$ for all sufficiently small, positive $\epsilon$. This shows that if you start with initial value $x_0 > x$, but close to $x$, since $f(x_0) > 0$ we will have that the ODE forces the particle to increase its value of $x$, and move away from the fixed point. If you start with $x_0 < x$, but close to $x$, the ODE will now force the particle to decrease its value of $x$, and move away from the fixed point. Hence if $f'(x) > 0$ we say that the fixed point is unstable.
Similarly you can work it out to show that if $f'(x) < 0$, the fixed point is stable.
One should note that the test of $f'(x) <0$ and $f'(x) > 0$ are merely sufficient to guarantee that the fixed point is stable or not. They are not necessary. That is, it is possible to have stable and unstable fixed points where $f'(x) = 0$. So in general stability/instability does not require the derivative of $f$ to have a sign, while if the derivative $f$ has a sign, then the fixed point must be accordingly stable/unstable.