I'm trying to come up with an example of a nonconstant function $f:\mathbb{R}^2\to\mathbb{R}$ such that $f(x, y)=f(A(x), A(y))$ for any affine transformation of the form $A(z) = az+b$ where $a>0, b\in\mathbb{R}$. Intuitively, I don't think it exists, but how do I prove whether it exists?
is there a theorem that says, or something to the effect of, the number of invariants cannot be greater than or equal to that of variables (two, $a$ and $b$ in this case)?
First, taking $a = 1$, we have that
$$f(x, y) = f(x + b, y + b)$$
for any $b$. Setting $b = -y$ gives $f(x, y) = f(x - y, 0)$ so if we set $g(z) = f(z, 0)$ then $f(x, y) = g(x - y)$; that is, $f$ is invariant under transformations of the above form iff it is a function of $x - y$ only.
Next, taking $b = 0$, we have that
$$f(x, y) = f(ax, ay) = g(x - y) = g(a(x - y))$$
for any $a > 0$. This condition is satisfied iff $g(z) = g(az)$ for any $a > 0$. If $z = 0$ then this condition is always satisfied, and if $z \neq 0$ then setting $a = \frac{1}{|z|}$ gives that this condition is satisfied iff $g(z) = g(1)$ if $z > 0$ and $g(z) = g(-1)$ if $z < 0$.
Finally, every transformation of the form $A(z) = az + b$ is a composition of transformations of the above two forms (namely, first multiply by $a$, then add $b$), so $f$ is invariant under all of them iff it's invariant under the two kinds of transformations above. It follows that the space of possible $f$ is $3$-dimensional, given by the $3$-parameter family of functions
$$\boxed{ f(x, y) = \begin{cases} g(0) & \text{ if } x = y \\ g(1) & \text{ if } x > y \\ g(-1) & \text{ if } x < y \end{cases} }.$$
where $g(0), g(1), g(-1)$ are arbitrary real numbers. If we furthermore require that $f$ is continuous then $f$ must be constant.
Abstractly you are looking at the subject of group actions in group theory. If a group $G$ acts on a set $X$ and you want to study functions $f : X \to Y$ invariant under this group action, meaning that $f(gx) = f(x)$ for all $g \in G$, then the general pattern is that this is the same thing as studying functions $f : X/G \to Y$ from the quotient $X/G$ of $X$ by the group action. In this case $X = \mathbb{R}^2$ and $G$ is the group of affine transformations $z \mapsto az + b, a > 0$, and the above argument essentially computes that the quotient $X/G$ consists of exactly $3$ points. These correspond to the orbits of the group action, which correspond to the three cases $x = y, x > y, x < y$ above; what this is saying is that these three conditions are each preserved by the action of $G$.
is there a theorem that says, or something to the effect of, the number of invariants cannot be greater than or equal to that of variables (two, $a$ and $b$ in this case)?
I'm not sure what you have in mind here, but a general idea about quotients by group actions is that if $X$ and $G$ both have dimensions in a suitable sense (e.g. if they are both manifolds) then if the action of $G$ on $X$ is "sufficiently nice" we expect $\dim X/G = \dim X - \dim G$ in some sense. This happens in the above case; $X = \mathbb{R}^2$ is $2$-dimensional and so is $G$, and the quotient is $3$ points which is $0$-dimensional.
