Given $f:\mathbb{R}^2\to\mathbb{R}, \ f(x,y) =axy+2(x+y)+b, \ a,b\in\mathbb{Z}$, where $f$ is a monoid, how many elements have an inverse? 
Given $f:\mathbb{R}^2\to\mathbb{R}, \ f(x,y) =axy+2(x+y)+b, \ a,b\in\mathbb{Z}$, where $f$ is a monoid, how many elements have an inverse?

How does one start solving such a problem? I'm not sure at all what the first step even is. What I tried is to find the neutral element $e$, such that $f(x,e)=f(e,x)=x$ but this yields $\frac{-x-b}{ax+2}$ and doesn't help at all. How should I begin approaching this?
 A: In fact, the way the question is asked is a little biased.
It is said that internal operation defined by $f$ defines a monoid, i.e., an associative law with a neutral element ; associativity looks "granted", but this is not at all the case. There are very few values of $a$ and $b$ for which the law is associative.
Indeed, let us compute :
$$ \begin{cases}f(f(x,y),z)&=&4x+4y+(2+ab)z+A\\ f(x,f(y,z))&=&(2+ab)x+4y+4z+A \end{cases} $$ $$\text{with} \ \ A=2a(xy+yz+zx)+a^2xyz$$
Associativity means :
$$\forall x,y,z, \ \ f(f(x,y),z) = f(x,f(y,z))$$
the coefficients of $x,y,z$ must be the identical ; therefore $2+ab=4$ giving $ab=2$ ending up with 4 possibilities :
$$(a,b)=(-2,-1), \ (-1,-2), \ (2,1), \ (1,2)$$
Can you now treat the existence of a neutral element in the different cases ?
Remark : Let us make a focus on the last case $(a,b)=(1,2)$. Let us write function $f$ :
$$f(x,y) = xy+2(x+y)+2 = (x+2)(y+2)-2$$
otherwise said, with $\varphi(x)=x+2$ :
$$x \star y = \varphi^{-1}(\varphi(x) \times \varphi(y)),$$
a relationship that will be clearer when written under the form of a diagram, displaying a so-called "transfer" the of ordinary product denoted by $\times$ and the "special" product denoted by $\star$.
$\require{AMScd}$
\begin{CD}
(x,y) \in \Bbb{R}^2 @>{\star}>> x \star y \in \Bbb{R} \\
@V{\varphi}VV @AA{\varphi^{-1}}A \\
(\varphi(x),\varphi(y)) \in \Bbb{R}^2 @>>{\times}> \varphi(x) \times \varphi(y)\in \Bbb{R}
\end{CD}
