Is there a foolproof method to calculate the inverse of a group homomorphism? For example, take the group $\langle a,b\mid a^{-1}b^2ab^{-3}\rangle$, and the homomorphism given induced by the map $a \rightarrow a$, $b \rightarrow b^2$. Is there a method that will  let you calculate the inverse of a homomorphism of a finitely presented group?
 A: Well, there's a simple brute-force algorithm that always works.  Let us suppose $f:\langle A\mid R\rangle\to\langle B\mid S\rangle$ is an isomorphism and we have an expression for $f(a)$ as a word in elements of $B$ for each $a\in A$.  To find the inverse, take every single word $w$ in $A$ and attempt to use the relations in $S$ to reduce $f(w)$ to some element of $B$.  Dovetailing these computations over all words $w$, you will eventually find for each $b\in B$ some $w_b$ such that $f(w_b)=b$.  Then, the inverse is given by $f(b)=w_b$.
Note that there is not any algorithm that determines whether a homomorphism between two finitely presented groups is an isomorphism.  Indeed, there is no algorithm that determines whether a finitely presented group is trivial, and so in particular there is no algorithm that determines whether the trivial homomorphism between two finitely presented groups is an isomorphism.
A: A group is Hopfian if every surjective homomorphism is an isomorphism (that is, surjective implies injective). This property was discussed at length in this old question.
Your map is not injective, and hence has no inverse. As Derek Holt points out in the comments, this is the standard example of a finitely presented non-Hopfian group.
The Baumslag-Solitar group $\operatorname{BS}(2, 3)=\langle a, b\mid a^{-1}b^2ab^{-3}\rangle$ is not Hopfian. To see this, consider the map you give in your question, $\phi: a\mapsto a$, $b\mapsto b^2$. Then $\phi(ba^{-1}bab^{-1}a^{-1}b^{-1}a)=1$ (this is easy to see) but $ba^{-1}bab^{-1}a^{-1}b^{-1}a\neq1$ (this is harder to see - it follows from Britton's Lemma for HNN-extensions).
