What are the minimum constraints on the domain and range of a function in order for it to be differentiable? For example if $S$ is a set with two elements $a$ and $b$, then you could define:
$f : \mathbb{R} \to S, s.t. f(x) = a, \forall x \in \mathbb{R}$
Intuitively, since f is a constant function, it seems it ought to be be differentiable, but is the constant value of $f'$ going to be $a$ or $b$? Obviously some structure (e.g. Group or Semigroup) is needed on $S$ in order to pick one.
 A: There is no such thing as the minimal constraint necessary to define the concept of differentiability. Several theories exist that go way beyond smooth manifolds, but in different (though overlapping) directions. Roughly speaking, one can axiomatize the Mean Value Theorem, or the first-order Taylor expansion, or the Leibniz product rule, etc. The survey Nonsmooth Calculus by Heinonen presents a readable overview of this area, though reading it requires a solid background in analysis.  
A: As Qiaochu points out, you should look up differentiable manifolds. There the derivatives are not functions to the manifold $M$ itself but rather into the tangent space bundel, which is a vector space "attached" to each point of $M$.
The tangent space at $p\in M$ can be constructed as equivalence classes of curves through $p$, where we declare those curves equivalent which we wish to be in the following sense.
If $\gamma\colon \mathbb R\to M$ is any curve with $\gamma(0)=p$, the derivative at $t=0$ as element in the tangent space $T_p$ should describe the "speed" at which we pass through $p$.
If we replace $\gamma$ with $t\mapsto\gamma(v t)$, we therefore wish the "speed" to be multiplied by $v$. Hence at least multiplication with real numbers should be possible in $T_p$. This does not make $T_p$ a vector space yet because we lack addition.
Also, we have not made up our mind yet, which curves to call differentiable in the first place.
In the end, the most natural setting to have a sufficiently rich theory of differentiability is indeed that of differentiable manifolds, i.e. spaces that "look like" copies of $\mathbb R^n$ smoothly glued together.
