Third order term in Taylor Series

Question

What is the third order term in the Taylor Series Expansion? I know it will just be third order partial derivatives but I want to know how is it expressed in a compact Matrix notation. For instance Jacobian for first order, Hessian for second order partial derivatives.

In other words, what is the third order term in the equation below? Thanks for your help!

Answer 1

Well, $f: \mathbb{R}^n \rightarrow \mathbb{R}^n$ has multivariate Taylor expansion at $x+h$ centered at $x$ of: $$ f(x+h) = f(x)+\sum_{i=1}^n \frac{\partial f}{\partial x_i}h_i +\frac{1}{2}\sum_{i,j=1}^n \frac{\partial^2 f}{\partial x_i\partial x_j}h_ih_j+\frac{1}{6}\sum_{i,j,k=1}^n \frac{\partial^3 f}{\partial x_i\partial x_j\partial x_k}h_ih_jh_k+ \cdots$$ Here we could define $T: \mathbb{R}^n \times \mathbb{R}^n \times \mathbb{R}^n \rightarrow \mathbb{R}^n $ by its values on the basis elements $e_i,e_j,e_k$ (unit-vector basis for Cartesian coordinates in $n$-dimensions) $$ T_{ijk}=\sum_{i,j,k=1}^n \frac{\partial^3 f}{\partial x_i\partial x_j\partial x_k} $$ The question is, can we write the formula $\sum_{i,j,k=1}^nT_{ijk}h_ih_jh_k$ as a matrix multiplication of some sort... I suggest we consider $\mathbb{R}$-valued $f$ as a starting point.

(with the proper conventions set forth there is a way to write this as a matrix multiplication, but, I think in the process of doing such we lose track of the manifest trilinearity, I'm not well-versed in the needed notation at this point in time so I'll leave my answer here as it stands since expressing this as a matrix multiplication on an abstract space is not that interesting, well, at least to me today)

Answer 2

$\renewcommand{\Re}{\mathbb{R}}$Often Taylor expansions of functions $f:\Re^n\to\Re$ at a point $x$ are meant along a given direction $d\in\Re^n$. This facilitates a lot out understanding even for first-order expansions.

Let $f\in\mathcal{C}^3$ and define a function $\phi:\Re\to\Re$ given by $\phi(\tau) = f(x+\tau d)$; this is like a slice of $f$: it describes $f$ along the line $x+\tau d,\tau\in\Re$. Then $\phi$ is three times continuously differentiable and the third order expansion of $\phi$ about $\tau=0$ is

$$\begin{aligned} \phi(\tau) = \phi(0) + \tau \phi'(0) + \tfrac{t^2}{2!}\phi''(0) + \tfrac{t^3}{3!}\phi'''(0) + o(t^3). \end{aligned}$$

But $\phi'(0)$ is related to the directional derivative of $f$ at $x$ along the direction $d$ which is

$$\begin{aligned} \phi'(0) &= \lim_{h\to 0}\frac{\phi(h) - \phi(0)}{h}\\ &= \lim_{h\to 0}\frac{f(x+h d) - f(x)}{h}\\ &= \langle \nabla f(x), d \rangle \end{aligned}$$

Let us denote this by $\nabla_{d}f(x)$.

Similary, $\phi''(0)$ can be interpreted as the directional Hessian of $f$ at $x$ along the directions $d$ and $d$, that is

$$\begin{aligned} \phi''(0) = \langle \nabla^2 f(x)d, d \rangle \end{aligned}$$

Let us denote this by $\nabla^2_{d,d}f(x)$.

The term $\phi'''(0)$ - the "Tressian" as James S. Cook jokingly put it - is more difficult to represent. Indeed, it will be a tensor as Oren explained. However, we are merely interested in the "directional Tressian" of $f$ at $x$ along directions $d$, $d$ and $d$. This construct is actually used in the context of convex optimization theory and in particular the theory of self-concordant functions and is denoted by $\nabla^3_{d,d,d}f(x)$ and we may write

$$\begin{aligned} \nabla^3_{d,d,d}f(x) = \langle \nabla^3 f(x)[d]d, d \rangle \end{aligned}$$

where $\nabla^3 f(x)$ is the third-order gradient of $f$ at $x$ which, in my opinion, is best understood via its directional variant:

$$\begin{aligned} \nabla^3f(x)[d] = \lim_{h\to 0} \frac{\nabla^2 f(x+\alpha d) - \nabla^2 f(x)}{h} \end{aligned}$$

Here $\nabla^3f(x)[d]$ is a matrix - it is a directional Hessian. Essentially, $\nabla^3f(x)[d]$ describes how the Hessian of $f$ changes at $x$ along the direction $d$.

Answer 3

You can take a look at my paper at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2364989. I use a mapping from tensor to matrix that enables to write a fifth order Taylor series in matrix notation. Take for example the third order term. If you have a tensor $f_{xxx}$ of dimensions $n\times n\times n\times n$ whose $m,i,j,k$ element is the derivative of the $m$'th entry of $f$ wrt the $i,j,k$ entries of $x$, you can use the reshape function to reshape it into a matrix of dimensions $n\times n^3$. Then, $\frac{1}{6}g_{xxx}\left( x\otimes x\otimes x\right)$ is the third order term of the Taylor series in matrix notation.

Answer 4

I actually found a way to express it in matrix-vector notation.

Let $\mathbf{x}\in \mathbb{R}^n$ then the third order term you're looking for is $$\frac{1}{3!}\mathbf{x^T(x^T\mathfrak{D})x},$$

where $\mathfrak{D}$ is a $n\times n\times n$ tensor which looks like a $n\times 1$ vector with $n\times n$ matrices as its elements that correspond to the $n^3$ possible 3rd order partial derivatives. Note that for functions that satisfy Schwarz' theorem, the following is true and thus the $n\times n$ matrices are symmetric: $(\partial_{ij})_k=(\partial_{ji})_k$ and $(\partial_i)_{jk}=(\partial_i)_{kj}$.

Note that $\mathbf{x^T\mathfrak{D}}$ should be the product that should yield $\sum_i\mathfrak{D_i}x^i$, $\mathfrak{D_i}$ being every one of those $n\times n$ matrices.

$\mathbf{EDIT}$:

I actually found an easier notation:

$$\mathbf{x^{T}}\mathbf{\mathfrak{D}x}= \begin{pmatrix} x^1 & \dots & x^n \end{pmatrix} \begin{pmatrix} \mathbf{f_{11}\circ x} & \dots & \mathbf{f_{1n}\circ x}\\ \vdots & \ddots & \vdots\\ \mathbf{f_{n1}\circ x} & \dots & \mathbf{f_{nn}\circ x} \end{pmatrix} \begin{pmatrix} x^1\\ \vdots\\ x^n \end{pmatrix},$$ where $\mathbf{f_{ij}}=(f_{ij1},\dots,f_{ijn})$ and the indices on the $f$s mean the partial derivatives wrt the $i$th or the $j$th coordinate.

$\mathbf{EDIT\ 2}:$

It's quite obvious but if you want you can express the matrix like this, which makes more sense, \begin{pmatrix} \mathbf{\nabla}f_{11}\mathbf{\circ x} & \dots & \mathbf{\nabla}f_{1n}\mathbf{\circ x}\\ \vdots & \ddots & \vdots\\ \mathbf{\nabla}f_{n1}\mathbf{\circ x} & \dots & \mathbf{\nabla}f_{nn}\mathbf{\circ x} \end{pmatrix}