Third order term in Taylor Series What is the third order term in the Taylor Series Expansion? I know it will just be third order partial derivatives but I want to know how is it expressed in a compact Matrix notation. For instance Jacobian for first order, Hessian for second order partial derivatives.
In other words, what is the third order term in the equation below? Thanks for your help!

 A: Well, $f: \mathbb{R}^n \rightarrow \mathbb{R}^n$ has multivariate Taylor expansion at $x+h$ centered at $x$ of:
$$ f(x+h) = f(x)+\sum_{i=1}^n \frac{\partial f}{\partial x_i}h_i +\frac{1}{2}\sum_{i,j=1}^n \frac{\partial^2 f}{\partial x_i\partial x_j}h_ih_j+\frac{1}{6}\sum_{i,j,k=1}^n \frac{\partial^3 f}{\partial x_i\partial x_j\partial x_k}h_ih_jh_k+ \cdots$$
Here we could define $T: \mathbb{R}^n \times \mathbb{R}^n \times \mathbb{R}^n \rightarrow \mathbb{R}^n $ by its values on the basis elements $e_i,e_j,e_k$ (unit-vector basis for Cartesian coordinates in $n$-dimensions)
$$ T_{ijk}=\sum_{i,j,k=1}^n \frac{\partial^3 f}{\partial x_i\partial x_j\partial x_k} $$
The question is, can we write the formula $\sum_{i,j,k=1}^nT_{ijk}h_ih_jh_k$ as a matrix multiplication of some sort... I suggest we consider $\mathbb{R}$-valued $f$ as a starting point. 
(with the proper conventions set forth there is a way to write this as a matrix multiplication, but, I think in the process of doing such we lose track of the manifest trilinearity, I'm not well-versed in the needed notation at this point in time so I'll leave my answer here as it stands since expressing this as a matrix multiplication on an abstract space is not that interesting, well, at least to me today)
A: $\renewcommand{\Re}{\mathbb{R}}$Often Taylor expansions of functions $f:\Re^n\to\Re$ at a point $x$ are meant along a given direction $d\in\Re^n$. This facilitates a lot out understanding even for first-order expansions.
Let $f\in\mathcal{C}^3$ and define a function $\phi:\Re\to\Re$ given by $\phi(\tau) = f(x+\tau d)$; this is like a slice of $f$: it describes $f$ along the line $x+\tau d,\tau\in\Re$. Then $\phi$ is three times continuously differentiable and the third order expansion of $\phi$ about $\tau=0$ is
$$\begin{aligned}
\phi(\tau) = \phi(0) + \tau \phi'(0) + \tfrac{t^2}{2!}\phi''(0) + \tfrac{t^3}{3!}\phi'''(0) + o(t^3).
\end{aligned}$$
But $\phi'(0)$ is related to the directional derivative of $f$ at $x$ along the direction $d$ which is
$$\begin{aligned}
\phi'(0) &= \lim_{h\to 0}\frac{\phi(h) - \phi(0)}{h}\\
         &= \lim_{h\to 0}\frac{f(x+h d) - f(x)}{h}\\
         &= \langle \nabla f(x), d \rangle
\end{aligned}$$
Let us denote this by $\nabla_{d}f(x)$.
Similary, $\phi''(0)$ can be interpreted as the directional Hessian of $f$ at $x$ along the directions $d$ and $d$, that is
$$\begin{aligned}
\phi''(0) = \langle \nabla^2 f(x)d, d \rangle
\end{aligned}$$
Let us denote this by $\nabla^2_{d,d}f(x)$.
The term $\phi'''(0)$ - the "Tressian" as James S. Cook jokingly put it - is more difficult to represent. Indeed, it will be a tensor as Oren explained. However, we are merely interested in the "directional Tressian" of $f$ at $x$ along directions $d$, $d$ and $d$. This construct is actually used in the context of convex optimization theory and in particular the theory of self-concordant functions and is denoted by $\nabla^3_{d,d,d}f(x)$ and we may write
$$\begin{aligned}
\nabla^3_{d,d,d}f(x) = \langle \nabla^3 f(x)[d]d, d \rangle
\end{aligned}$$
where $\nabla^3 f(x)$ is the third-order gradient of $f$ at $x$ which, in my opinion, is best understood via its directional variant:
$$\begin{aligned}
\nabla^3f(x)[d] = \lim_{h\to 0} \frac{\nabla^2 f(x+\alpha d) - \nabla^2 f(x)}{h}
\end{aligned}$$
Here $\nabla^3f(x)[d]$ is a matrix - it is a directional Hessian. Essentially, $\nabla^3f(x)[d]$ describes how the Hessian of $f$ changes at $x$ along the direction $d$.
A: You can take a look at my paper at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2364989. I use a mapping from tensor to matrix that enables to write a fifth order Taylor series in matrix notation. Take for example the third order term. If you have a tensor $f_{xxx}$ of dimensions $n\times n\times n\times n$ whose $m,i,j,k$ element is the derivative of the $m$'th entry of $f$ wrt the $i,j,k$ entries of $x$, you can use the reshape function to reshape it into a matrix of dimensions $n\times n^3$. Then, $\frac{1}{6}g_{xxx}\left( x\otimes x\otimes x\right)$ is the third order term of the Taylor series in matrix notation.
