It is not that they aren't rigorous, it's that calculus books, as usual, don't necessarily take the care to make the relevant distinctions to make it fully rigorous. They can be made rigorous.
One thing I'd argue is that the "instantaneous rate of change" is something which can be defined as formally equivalent to, but conceptually distinct from, the derivative, with the derivative being more general. A derivative, in the case of a function of a real variable, is a certain quantity that characterizes the local behavior of such a function around an input point and how it responds to small changes in that input or, better, how its output differs when considering input values slightly different from a particular input and compared against the value it attains at that particular input.
The reason I say this is because the concept of a "rate of change" implicitly presumes a flow of time, and not all derivatives involve time.
The derivative of $f$ at $a$ is defined by
$$f'(a) := \lim_{\Delta x \rightarrow 0} \frac{f(x + \Delta x) - f(x)}{\Delta x}$$
as you already know.
But now for the instantaneous rate of change. Parsing that term, we'd ideally want to say that, to make the intuition in it rigorous, we should define both what a "rate of change" is and, moreover, what it means for that rate to be "instantaneous".
So how do we do that? Analyzing the term further, we see we need to define "change" and "rate". The change - before we get to "rate of" - of a temporally-varying quantity from time $t_1$ to time $t_2$, given as a function $f$ of time, is thus defined by
$$\text{Change in $f(t)$ from time $t_1$ to time $t_2$} := f(t_2) - f(t_1)$$
i.e. change is just subtraction (difference). The rate of change then, is the ratio of two changes (note that the "change in time" can be understood as the change of the identity function of time, so we don't need another definition):
$$\text{Rate of change of $f(t)$ from time $t_1$ to time $t_2$} := \frac{\text{Change in $f(t)$ from time $t_1$ to time $t_2$}}{\text{Change in time from $t_1$ to $t_2$}}$$
from which we can see that
$$\text{Rate of change of $f(t)$ from time $t_1$ to time $t_2$} = \frac{f(t_2) - f(t_1)}{t_2 - t_1}$$
So what then is the instantaneous rate of change? Logically, it is the rate of change at a single instant, i.e. when $t_1 = t_2 = t_a$ at a particular instant $t_a$. However, we cannot achieve that with the above definition because we get a division-by-0 error. Instead, what we must do is use a limit to fill it in - in particular, we should take the following two-dimensional limit:
$$\text{Instantaneous rate of change of $f$ at $t_a$} := \lim_{(t_1, t_2) \rightarrow (t_a, t_a)} \frac{f(t_2) - f(t_1)}{t_2 - t_1}$$
where that we only consider points $t_1$ and $t_2$ such that both $t_1 \le t_a \le t_2$, i.e. the intervals of change "bracket" our desired point $t_a$, and $t_1 \ne t_2$. Then we have
Theorem: If the IRoC of $f$ exists at $a$, then it equals $f'(a)$.