Has this operator a name? $\alpha_g(f)=g \ominus f=f\circ g \circ f^{\circ (-1)}$ In my free time plaing/studing functions and function composition (as amateur mathematician) I found a procedure with intresting properties that I called "function of functions" $\alpha_g(f)$  (Really I defined a binary operation $\alpha_g(f)=g \ominus f$ and  $\ominus:\mathcal H\times \mathcal H\rightarrow \mathcal H $ at the beginning).
I have already asked about the behaviour of this function of functions $\alpha_g(f)$ (when is well defined) and the functions that it generates in this other question:

Sets $f_n\in A_f$ where $f_{n+1}=f_n \circ S \circ f^{\circ (-1)}_n$
and operator $\alpha(f_n)=f_{n+1}$

and an user told me that what I'm talking about is an operator between two spaces of functions (real valued in that case) but he did not give me informations about what is this operator, its name and if it is know.
On SE Mathematics Meta here another user told me that was better to open a new question with just one question in order to be more specific, this is that question.
I redefine my question here and explain what I'm talkling about.
$\mathcal H$ is a set of functions $f:X \rightarrow X$ where we can always define this operator (exist an inverse function) $\alpha :\mathcal H\rightarrow \mathcal H$
$$\alpha_g(f)=f\circ g \circ f^{\circ (-1)}$$
This operator has some intresting propeties (I find them interesting) but I don't know If Is well studied (maybe is not so interesting for the mathematics) so my questions are:

$1)$ Has this operator a name? Some references and links?
$2)$ Is the Operator theory that study objects like this? If not how is called that field of mathematics?

I want to find something to study because I like very much this operator and I want to know more about it: for example when it is defined, and other results that are beyond my knowledge.
Opinions
I read the all wikipedia page about operators, but I can' find it. It talks about normed,
bounded, linear and nonlinear operators, and how is importand to have some structure
(norm, metric or topology) on the domain of the operator... but I can't find what I need.
At the moment the only thing maybe I've understood is that $\alpha$ is nonlinear when we define it on the set of real function (with the usual addition) because $\alpha_S(f+g)\neq \alpha_S(f)+\alpha_S(g)$ ($S$ is the successor function).
Anyways I found some intresting propeties that holds sometimes (if the functions are invertible i think), that maybe make you able to recognize this operator.
$\alpha_f(f)=f$ ($f$ invertible?)
$\alpha_h(f+g)=(\alpha_h(f)\circ f \circ (f+g)^{\circ (-1)})+(\alpha_h(g)\circ g \circ (f+g)^{\circ (-1)})$
and in general if $\alpha_h(g)=h\ominus g$
$\alpha_h(f \circ g)=\alpha_h(g)\ominus f$
I don't claim that these affarmations are correct (with some restrictions are corrects maybe), I just noticed something that maybe can help to make the question more clear.
Thanks in advance and sorry ofr the long post and errors.
 A: The notation really should be something like $\alpha_f(g)$. This is conjugation by $f$ in the monoid of functions under composition. Its two most important properties are that
$$\alpha_f(g \circ h) = \alpha_f(g) \circ \alpha_f(h)$$
($\alpha_f(-)$ is a homomorphism from the monoid of functions to itself) and
$$\alpha_{f \circ g}(h) = \alpha_f(\alpha_g(h))$$
($\alpha_{-}$ is a homomorphism from the group of invertible functions to the automorphism group of the monoid of functions; these are the inner automorphisms). Conjugation is extensively studied in many parts of abstract algebra, such as group theory. Probably the most concrete is conjugation of matrices; see matrix similarity. Another relatively concrete case is conjugation of permutations. 
For historical reasons the term "operator" in mathematics often has a much more specific meaning than it sounds related to functional analysis. It wasn't the right keyword to search for. 
In computer science a "function of functions" is called a higher-order function. This terminology is uncommon in mathematics, though (since we generally regard all sets, including sets of functions, on an equal footing). 
A: As far as your functions f and g can be expressed by polynomials or power series one can translate that "algebra" into the language of matrices, namely "Carleman-matrices" (or "Jabotinsky-matrices" which seem to cover the field in even more generality).
Once you have your problem reformulated as a relation between Carleman-matrices, say $F$ for function $f(x)$ and $G$ for function $g(x)$ you have $F G F^{-1}$ for $ f \circ g \circ f^{\circ -1 }$ and so on. If $G$ is diagonal, then $F$ would be the eigenvector-matrix of the Carlemanmatrix $H = F \cdot G \cdot F^{-1}$ and re-tranlated $f(x)$ is the Schroeder-function for $h(x) = f \circ g \circ f^{\circ -1} (x)$ - and this opens an extremely wide and interesting field in the area of functional iteration.
It might be allowed to remark that I've been involved in tetration for some years now and entered that area stumbling incidentally on that Carleman-matrices; for a furtherly motivating discussion you might possibly be interested in the (amateurish) introductional observations on functional composition using Carlemanmatrices on my website (here the index)
