Generalizing a function that operates on functions I've not studied much beyond Calculus so my notation and terminology may be rough/incorrect.
I've got a 'function' $\mathfrak F$ that uses other functions and goes like this:
$$
    \mathfrak F(f,n) = D_x^nf \\
    f:\Bbb R \rightarrow \Bbb R \\
    n \in \Bbb Z
$$
Some examples:
$$
    f(x) = x^2 \\
    \mathfrak F(f,2) = D_x^2x^2 = 2 \\
    \mathfrak F(f,-3)= D_x^{-3}x^2 = \frac{1}{60}x^5 + \frac{C_1}{2}x^2+C_2x+C_3
$$
My questions are:
1. What's the proper notation for this?
2. How can I write this so that $n \in \Bbb R$ or even $n \in \Bbb C$?
3. How can I write this so that $f$ is multi-variable?
Finally,
4. Would this 'function' be differentiable along $n$ ($n \in \Bbb R$)?
 A: *

*Your notation is fine, but incomplete in two ways. Firstly you have not specified a domain and a codomain for the operator $\mathscr{F}$. For the domain we can take the smooth functions $\mathcal{C}^{\infty}(X,Y)$, where $X$, $Y \subseteq \mathbb{R}^d$ are open.
However, for negative $n$ this will not work for the codomain, as the constants you introduce do not specify a unique smooth function. Hence you need to make a choice of constants here to make things uniform. For instance you could set them all to zero. Secondly $\mathscr{F}(f,n) \in \mathcal{C}^{\infty}(X,Y)$ is a function. Consequently one should write $\big(\mathscr{F}(f,n)\big)(x)$ if you specify with a variable $x$ on the right hand side.  

*This has been extensively studied under the slogan fractional calculus. One way of defining those things is to use Fourier-Transformations, which turn differentiation into multiplication operators. Then one only has to understand exponentiation by complex numbers, which itself is not completely harmless. To make all operations well defined, one usually restricts oneself to a smaller function space than $\mathcal{C}^{\infty}(X,Y)$, such as the Schwartz space. The latter is nice, because the Fourier-Transform is an isomorphism of the Schwartz space. Note that this is not the only way of doing about extending the 

*Generalizing all of the above to higher dimensions, i. e. more variables, is straightforward by the use of multiindex notation. Typically the additional bookkeeping turns out to be harder than the mathematical generalizations needed.

*Talking about smoothness requires differentiable structures on the domain and codomain of your operator. The involved function spaces are infinite dimensional, but can be given the structure of a Fréchet manifold. As the other answer stated, you should look into the notion of Differential operators for more about this. Note that there is quite a bit of differential geometry and functional analysis involved at this stage. And to your question: I do not know, but I assume one could define a fractional calculus by smooth extension of the involved operators after defining them for $n \in \mathbb{Q}$.

