Is this a field of study? Is there a name for an equation that takes the following form?
$$F(f(x),f^{-1}(x),x)=0$$
A nice example being
$$f(x)-f^{-1}(x)=0$$
because the solutions of this equation are their own inverses. WolframAlpha solved this problem, though I had to type the equation as $f(f(x))=x$.
Is there any nice method for solving such equations? How do you think WolframAlpha does it?
For instance, what if I wanted to solve
$$f(x)={f^{-1}(x)}^2+x$$
Which could also be written as
$$f(\sqrt{f(x)-x})-x=0$$
 A: Yes, basic functional equations theory. It is far more hit-or-miss than other more systematic fields. 
Resource recommendation: You  might start with something practical, like C Efthimiou's Introduction to Functional Equations, 
AMS,  ISBN: 978-0-8218-5314-6 online; and only then proceed to, e.g. J Aczel's Lectures on Functional Equations and Their Applications, ISBN: 9780486445236 , and on and on...
First, resolve the constraint, as you are doing, and neutralize inverses, as you already did. One cannot reverse-engineer W-Alpha, but it simply looks things up from a lookup list. See it gag, abjectly, on , e.g. , $f(f(x))=-x$.
It looked up $f(f(x))=x$, since this is the celebrated Babbage equation, solved in 1815. Note from the WP article that any workable functional conjugate of a solution, so the basic fractional solution given, produces further solutions... an infinity of them. W-Alpha throws a handful of illustrative examples to the impressionable and uninformed audience, and all but invites them to find "new" solutions.... supplant the 3 by a 5 ... etc. There may be sweet conjugacy orbits of $\Psi(x)$s which you may find practical for your purposes. 
Your second example equation is messy, and, since you are asking orientation questions, I might supplant it with a much easier one, instead,
$$
f(f(x)-x)=x .
$$
Note the fixed point $f(0)=0$. Consider the Taylor expansion around it,
$$
f(x)=ax+bx^2+cx^3+...
$$
Solve for a,b,c,... recursively, order by order in x. 
A particularly central functional equation utilizing functional conjugacy is Schröder's equation.
