Different type of function composition When I first heard about function composition in school, I also expected the graphs of the composed function to 'behave' like both simple functions that are composed.
Suppose we have f(x) = $x^2$ and $g(x) = \cos(x)$
The graph of $y = f(g(x))$ looks like in the first image:

But my instinct in school was to think that the graph looks like the second image:
Red curve represents f(x) = $x^2$ and the blue curve represents the result of the special composition $y = f(g(x))$.

Think of this special type of composition as molding/transposing the X-axis on the graph/curve of the outer function in the composition and then draw the graph of the inner function on previously modified coordinates plane.
In the example above think of the parabola $x^2$ as the X-axis and then draw $ \cos(x)$ around this new X-axis (which is no longer a straight line).
My questions are:

*

*Does this type of function composition have a name?

*Has it been studied?

*Are there any papers I can read on it?

*If yes, how can I search about them?

*I know that in this type of composition, the result might not be a function that can be written in explicit form $y=f(x)$, but when can it be?

As a final note, here is $\cos(x)$ composed with
$\frac{1}{6}\cos(6x)$. It looks nice, like the beginning of a fractal series.
Red curve represents $g(x) = \cos(x)$ and the blue curve represents the result of the special composition.

 A: First the bad news: I've spent significant time in or around a few different university math departments and I've never heard of anyone studying an operation like this. You can view that as a positive thing: you've probably invented an original concept! Now you just have to go write papers proving cool features of these compositions until other people get interested and it becomes a new field of math :)
The thinking does remind me of some differential geometry ideas like tangent spaces. You're basically trying to use the tangent space of the "main curve" as a coordinate system for the "offset curve".
Formula for the new composition
Now the good news: I can give a (somewhat messy) formula that implements your intuition. Specifically I'm focusing on this quote from your text: Think of this special type of composition as molding/transposing the X-axis on the graph/curve of the outer function in the composition and then draw the graph of the inner function on previously modified coordinates plane.
I'll start with a bunch of definitions. I want to create a new curve called $f \hat \circ g$, where $f$ is the "base function" that defines the high-level shape, and $g$ is the "inner function" that impacts the smaller-scale movements. This will need to be a parametric curve because in general the graphs you want are not single-valued functions.
View the graph of $f$ as a parametric curve: $\mathbf r(t) = (t, f(t))$. This has tangent vector $\mathbf r'(t) = (1, f(t))$, and by rotating 90 degrees we can get a normal vector $\mathbf n(t) = (-f(t), 1)$. Define $\mathbf N(t)$ to be the unit length normal vector, i.e. $\mathbf N(t) = \frac{\mathbf n(t)}{\|\mathbf n(t)\|}$.
Let $L(t)$ be the arc length of $\mathbf r$ on the interval from $0$ to $t$. Then to follow your intuition, we'll want to build each point on our final curve by combining information from $f(t)$ with an offset from $g(L(t))$. In particular, the final formula will be:
$$f \hat \circ g(t) = \mathbf r(t) + g(L(t)) \mathbf N(t)$$
To test the formula, I implemented the new composition in Mathematica and I plotted the same test cases you used in the question. The graphs came out looking extremely similar to your example graphs, so I think the test is a success. I'm including my code for completeness; even if you don't use Mathematica, I think you could type basically the same thing in any CAS or even Python if you wanted to play with this operation some more.
f[x_] := x^2
g[x_] := Cos[x]
n[t_] := {-f'[t], 1};
nn[t_] := n[t]/Norm[n[t]]
fog[t_] := r[t] + g[ArcLength[r[s], {s, 0., t}]]*nn[t]
ParametricPlot[fog[t], {t, -5, 5}]


f[x_] := Cos[x]
g[x_] := Cos[6 x]/6
n[t_] := {-f'[t], 1};
nn[t_] := n[t]/Norm[n[t]]
fog[t_] := r[t] + g[ArcLength[r[s], {s, 0., t}]]*nn[t]
ParametricPlot[fog[t], {t, -8, 8}]


Pretty cool!
