For example, one can apply $\cos x$ to number $a$ one time to get $\cos a$, two times $\cos \cos a$, three times $\cos \cos \cos a$, and so on. Is there a way to define fractional application for $\cos$? Or for any other function? Maybe exists general theory for that?
Obviously $f(x,n)$ defined as taking $g$ $n$ times of $x$ is a function $(\mathbb R,\mathbb N)\to\mathbb R$. Any extension of $f$ to $(\mathbb R, \mathbb Q)$, could be considered a way to apply $g$ a rational number of times, if you extend it to $\mathbb R^2$, you could consider it a definition of applying $g$ any (real) number of times.
Extending functions to a larger domain is often done, a very well-known example being the factorial function being extended from $\mathbb N$ to $\mathbb C$ by the gamma function. But considering it applying a function a rational/real/complex/... number of times isn't so common, there's rarely any thing to be gained from viewing it that way.
In the specific case of $\cos$, I don't recall ever seeing such an extension, but we could define one, by saying that for any non-natural number of applications the result is $0$. It's not interesting but now we have a way to apply $\cos$ any number of times we want.