There is an inconsistency in how you are treating your symbols between these two equations:
$$h'(t) = af(t) - bh(t)$$
$$0 = af(t) - bh(t)$$
Note that in the first, you establish that the image of a particular value $t$ under the function $h'$, is equal to some algebraic combination of the images of this particular $t$ under two different functions, $f$ and $h$.
In the second equation, you continue to treat $f$ and $h$ as functions. However, since you have asserted that $h'(t) = 0$, we are now saying that for some particular value of $t$ in the domain, call it $t_0$, that $$h'(t_0)= 0 = af(t_0) - bh(t_0)$$
Rewriting it this way we can see that the RHS is reduced to a value. It is the image of $t_0$ under the function $h'$.
In other words, $\frac{a}{b} \frac{d}{dt} f(t_0) = \frac{d}{dt} h(t_0) = 0$. This is not the same as the general conclusion $\frac{a}{b} \frac{d}{dt} f(t) = \frac{d}{dt} h(t) = 0$. In the former we are taking the derivative with respect to a value, in the latter it is the derivative with respect to a function.