The equations that you have there are called the "relative error" equations. They give you a numerical estimate of the error of a quantity normalized against the scale of the quantity. That measurement is commonly used in Newton's Method to know how many times the routine needs to be applied by stopping once the required working precision has been reached.
The question you have is this: "Do I measure relative to the 'ideal' quantity or the 'approximate' quantity?" In the realm of numerical methods, you usually don't have the 'ideal' quantity, or you wouldn't be trying to solve for it. :) What you do is take the last two values of the quantity for the iterative procedure (such as Newton's Method), and measure the relative error that way, with the 'ideal' value being $x$, and the 'approximate' value being $\hat x$. This is done because we assume that the method is convergent, that given an infinite working precision, and infinite iterations, it will give us exactly the right answer. Thus, the idea is that the answers should be getting progressively closer to (hence, "converging") to the right answer. However, when choosing the denominator, if your relative error is very small, the difference in value of the two fractions you have shown is meaningless. What you are really concerned about is how big the difference $\Delta x=x-\hat x$ is relative to either of the values.