The basic problem is that the adjoint maps need not be injective. That is, the extra group relations may collapse all our part of your group into something much smaller than the monoid.
The right adjoint takes monoid elements to units. (This behavior is analogous to constructing the quotient of a ring with one of its maximal ideals.) For the multiplicative monoid $\Bbb{N}$, this takes everyone to 1. Reversing that is hopeless.
For the left adjoint, I refer to one of my favorite monoid actions: differentiation applied to a vector space of polynomials (of degree less than or equal to $N$ and coefficients in something nice, like $\Bbb{Z}$). The action is not injective. Given that, the inverse action of the single generator is not a map from a polynomial to a polynomial, but to an equivalence class of polynomials. I.e., just because I know the action of a monoid element does not mean the inverse action is well-definable. So you can construct a formal object containing those inverses, but there is no guarantee that these formal inverses are the right kinds of maps, or can be made so.
Maybe one can alter the set so as to make this work. Restrict the set? Partition the set into equivalence classes so that every monoid element is an injective map? But this isn't the construction you seem to be interested in.