According to your formula (d = |difference | and error = d/ 10), the possible values of error must lie between 0 and 1. Thus, we can use sinθ to represent the situation, i. e. sinθ = d/10.
In fact, you can build a “d vs θ” reference table [like when d = 1, the deviation is arcsin(0.1) = 5.74 degrees etc.]. Then, your “degree of the difference between their choices” is simulated by “degree (of an angle)”.
Another good thing about this system is that the guess need not be restricted to integers lying between 0 and 10.
As for the degree of similarity, you can use cosθ to represent it. But the formula need to be changed to $\sqrt (1 – d^2/(10^2))$ because $cosθ = sqrt(1 – sin^2θ)$.
Furthermore, you can redefine your formula as:-
D = first number chosen – second number chosen;
True Error = D/10; and
Sinθ = D/10
When θ lies between 0 and 90 (sinθ is positive), then you know that the first is greater than the second. Otherwise, the negative result is represented by an angle lying between 180 and 270 (or between 270 and 360).
As for the similarity, the above formula still applies except that there is no negative value and hence the result can only be represented by an angle lying between 0 and 90.
One can argue that the distinction between positive and negative similarities is ignored.
If still not happy, we can ‘borrow’ the negative sign from above and use the formula
$cosθ = – sqrt(1 – sin^2θ)$