As Qiaochu points out, you should look up differentiable manifolds. There the derivatives are not functions to the manifold $M$ itself but rather into the tangent space bundel, which is a vector space "attached" to each point of $M$.
The tangent space at $p\in M$ can be constructed as equivalence classes of curves through $p$, where we declare those curves equivalent which we wish to be in the following sense.
If $\gamma\colon \mathbb R\to M$ is any curve with $\gamma(0)=p$, the derivative at $t=0$ as element in the tangent space $T_p$ should describe the "speed" at which we pass through $p$.
If we replace $\gamma$ with $t\mapsto\gamma(v t)$, we therefore wish the "speed" to be multiplied by $v$. Hence at least multiplication with real numbers should be possible in $T_p$. This does not make $T_p$ a vector space yet because we lack addition.
Also, we have not made up our mind yet, which curves to call differentiable in the first place.
In the end, the most natural setting to have a sufficiently rich theory of differentiability is indeed that of differentiable manifolds, i.e. spaces that "look like" copies of $\mathbb R^n$ smoothly glued together.