How would the double derivative of $f:\mathbb{R}^N \to \mathbb{R}^M$ i.e., $f''$ look like? The derivative of $f:\mathbb{R}^N \to \mathbb{R}^M$ is of the form $f':\mathbb{R}^N \to \mathbb{R}^{M \times N}$. I'd like to know how the double derivative look like, i.e, how would $f''$ be ? It maps from $\mathbb{R}^N$ to which space?
PS : Please suggest some good references on this topic.
 A: Since you tagged (differential-geometry), let me give a slightly more geometric perspective. 
The "derivative" operation in the category of smooth manifolds is one that associates to a smooth map $f:M\to N$ between smooth manifolds $M$ and $N$ the map $Tf$ (or sometimes written as $\mathrm{d}f$ or $\nabla f$, I will not use the notation $\mathrm{d}f$ so as not to confuse with the exterior differentiation) between the smooth manifolds $TM$ and $TN$, where $TM$ denotes the tangent bundle of $M$. A specific property of the map $Tf$ is that restricted on the tangent space of a point $p\in M$, $Tf(p): T_pM \to T_{f(p)}N$ is a linear map between the tangent spaces of $M$ at $p$ and of $N$ and $f(p)$. 
This is already slightly different from the calculus definition. For linear spaces $X$ and $Y$, the calculus definition of a derivative gives that for $f:X\to Y$, $\nabla f: X\times X \to Y$ is linear in the second component. Formally we can think of the map $Tf$ as $(f,\nabla f)$ in this context, using that there is a canonical isomorphism between the tangent spaces $TX$ and $X\times X$ when $X$ is linear. 
Going back to the geometrical picture: naturally, the "second derivative" then is
$$ T(Tf) : T(TM) \to T(TN) $$
since the tangent bundles $TM$ and $TN$ are smooth manifolds in their own right, and $Tf$ is a smooth map in its own right. In the advanced calculus definition, given a linear space $X$ and a linear space $Y$, the second derivative is a map from $X\times X\times X \to Y$ that is bilinear in the second and third components. Analogous to the case before, we can try to formally identify 
$$ T(Tf) "=" ((f,\nabla f),\nabla(f,\nabla f)) \overset{?}{=} (f,\nabla f,\nabla f,\nabla^2 f) $$
While the co-domain has the correct number of dimensions, the domain however has some trouble: the domain of $T(Tf)$ is $TTM$, a manifold of $4m$ dimensions (assuming $M$ has $m$ dimensions). $\nabla^2 f$ however takes input as something only $3m$ dimensional! Furthermore, our intuitive notion of the second derivative from advanced calculus consists of taking two derivatives in two possibility different directions, where directions are interpreted as vectors along $M$, that is to say, two objects in $TM$. But as seen above, the notion of $T(Tf)$ requires considering objects in $TTM$. This (at the level of the second derivative) is the point where geometry departs from merely "calculus on manifolds", and this is the point where the notion of linear connection comes in. More precisely, to reconcile the notion with our usual notion of second derivatives, what we require is a way, when given $p\in M$, $v,w\in T_pM$, to identify the triple $(p,v,w)$ with an element of $T_{(p,v)}(TM)$. 
For more about these types of stuff, you may wish to consult:


*

*Kobayashi and Nomizu, Foundations of Differential Geometry

*Kobayashi, S. "Theory of connections" Ann. Mat. Pura Appl. (4), 1957, 43, 119-194

*Kolář, Michor, & Slovák, Natural operations in differential geometry
A: Let $X=\mathbb{R}^n$ and $Y=\mathbb{R}^m$, $f:X\to Y$ then $f': X\to L(X,Y)$ where $L(X,Y)$ is the space of (continuous) linear transformations. Since, by definition, we have $f''=(f')':X\to L(X,L(X,Y))$. Now if L$_2(X,Y)$ is the set of (continuous) bilinear functions $h:X\times X\to Y$ we can identify $L(X,L(X,Y))\cong L_2(X,Y)$ via the mapping $g(x_1)(x_2)\mapsto g(x_1,x_2)$. So the second derivative can be seen as a function $f'':X\to L_2(X,Y)$.
A reference for this would be Jost's "Postmodern analysis".       
A: Really $f' : \mathbb R^N \to \mathcal L(\mathbb R^N,\mathbb R^m)$ (linear maps). if $a \in \mathbb R^N$  then $f'(a)$  in a linear map from $\mathbb R^N$ to $\mathbb R^M$.
New, since $f''(a)=(f')'(a)$ , we can see that  $f'' : \mathbb R^N  \to \mathcal L(\mathbb R^N , \mathcal L(\mathbb R^N,\mathbb R^M))$ who is naturally isomorphic to  $\mathcal L_2((\mathbb R^N)^2, \mathbb R^M)$ : the set of bilinear maps frome $(\mathbb R^N)^2$  to $\mathbb R^M$.
