19
$\begingroup$

What is the third order term in the Taylor Series Expansion? I know it will just be third order partial derivatives but I want to know how is it expressed in a compact Matrix notation. For instance Jacobian for first order, Hessian for second order partial derivatives.

In other words, what is the third order term in the equation below? Thanks for your help!

Second Order Taylor Series Expansion

$\endgroup$
2
  • $\begingroup$ the third order term is connected with a trilinear, completely symmetric mapping. I'm not sure I've ever seen this written as a matrix multiplication. Fortunately, linear and quadratic forms permit a natural matrix representation, but, well, I await answers... How will we get a trilinear mapping on $\mathbb{R}^n$ from a multiplication with an $n \times n$ matrix? $\endgroup$ Commented Nov 8, 2013 at 15:19
  • $\begingroup$ Matrix notation would make it really easy to understand and program. If that is not possible, then is there an intuitive way to understand what all terms will be of the third order and what their coefficients will be? $\endgroup$
    – Innocent
    Commented Nov 8, 2013 at 15:27

5 Answers 5

10
$\begingroup$

Well, $f: \mathbb{R}^n \rightarrow \mathbb{R}^n$ has multivariate Taylor expansion at $x+h$ centered at $x$ of: $$ f(x+h) = f(x)+\sum_{i=1}^n \frac{\partial f}{\partial x_i}h_i +\frac{1}{2}\sum_{i,j=1}^n \frac{\partial^2 f}{\partial x_i\partial x_j}h_ih_j+\frac{1}{6}\sum_{i,j,k=1}^n \frac{\partial^3 f}{\partial x_i\partial x_j\partial x_k}h_ih_jh_k+ \cdots$$ Here we could define $T: \mathbb{R}^n \times \mathbb{R}^n \times \mathbb{R}^n \rightarrow \mathbb{R}^n $ by its values on the basis elements $e_i,e_j,e_k$ (unit-vector basis for Cartesian coordinates in $n$-dimensions) $$ T_{ijk}=\sum_{i,j,k=1}^n \frac{\partial^3 f}{\partial x_i\partial x_j\partial x_k} $$ The question is, can we write the formula $\sum_{i,j,k=1}^nT_{ijk}h_ih_jh_k$ as a matrix multiplication of some sort... I suggest we consider $\mathbb{R}$-valued $f$ as a starting point.

(with the proper conventions set forth there is a way to write this as a matrix multiplication, but, I think in the process of doing such we lose track of the manifest trilinearity, I'm not well-versed in the needed notation at this point in time so I'll leave my answer here as it stands since expressing this as a matrix multiplication on an abstract space is not that interesting, well, at least to me today)

$\endgroup$
2
  • 1
    $\begingroup$ Is there a name for $T_{ijk}$? $\endgroup$
    – jvriesem
    Commented Dec 11, 2015 at 17:41
  • 3
    $\begingroup$ @jvriesem It's the Tressian. No, just kidding, I know the second order piece is commonly referred to as the Hessian. I've not seen an in-depth analysis of just the third order piece. Anyway, I am not aware of a name. $\endgroup$ Commented Dec 12, 2015 at 3:22
7
$\begingroup$

$\renewcommand{\Re}{\mathbb{R}}$Often Taylor expansions of functions $f:\Re^n\to\Re$ at a point $x$ are meant along a given direction $d\in\Re^n$. This facilitates a lot out understanding even for first-order expansions.

Let $f\in\mathcal{C}^3$ and define a function $\phi:\Re\to\Re$ given by $\phi(\tau) = f(x+\tau d)$; this is like a slice of $f$: it describes $f$ along the line $x+\tau d,\tau\in\Re$. Then $\phi$ is three times continuously differentiable and the third order expansion of $\phi$ about $\tau=0$ is

$$\begin{aligned} \phi(\tau) = \phi(0) + \tau \phi'(0) + \tfrac{t^2}{2!}\phi''(0) + \tfrac{t^3}{3!}\phi'''(0) + o(t^3). \end{aligned}$$

But $\phi'(0)$ is related to the directional derivative of $f$ at $x$ along the direction $d$ which is

$$\begin{aligned} \phi'(0) &= \lim_{h\to 0}\frac{\phi(h) - \phi(0)}{h}\\ &= \lim_{h\to 0}\frac{f(x+h d) - f(x)}{h}\\ &= \langle \nabla f(x), d \rangle \end{aligned}$$

Let us denote this by $\nabla_{d}f(x)$.

Similary, $\phi''(0)$ can be interpreted as the directional Hessian of $f$ at $x$ along the directions $d$ and $d$, that is

$$\begin{aligned} \phi''(0) = \langle \nabla^2 f(x)d, d \rangle \end{aligned}$$

Let us denote this by $\nabla^2_{d,d}f(x)$.

The term $\phi'''(0)$ - the "Tressian" as James S. Cook jokingly put it - is more difficult to represent. Indeed, it will be a tensor as Oren explained. However, we are merely interested in the "directional Tressian" of $f$ at $x$ along directions $d$, $d$ and $d$. This construct is actually used in the context of convex optimization theory and in particular the theory of self-concordant functions and is denoted by $\nabla^3_{d,d,d}f(x)$ and we may write

$$\begin{aligned} \nabla^3_{d,d,d}f(x) = \langle \nabla^3 f(x)[d]d, d \rangle \end{aligned}$$

where $\nabla^3 f(x)$ is the third-order gradient of $f$ at $x$ which, in my opinion, is best understood via its directional variant:

$$\begin{aligned} \nabla^3f(x)[d] = \lim_{h\to 0} \frac{\nabla^2 f(x+\alpha d) - \nabla^2 f(x)}{h} \end{aligned}$$

Here $\nabla^3f(x)[d]$ is a matrix - it is a directional Hessian. Essentially, $\nabla^3f(x)[d]$ describes how the Hessian of $f$ changes at $x$ along the direction $d$.

$\endgroup$
3
  • $\begingroup$ Interesting. Where can we read more about this? $\endgroup$ Commented Nov 5, 2016 at 23:03
  • $\begingroup$ @JamesS.Cook I think a good reference is the book of Y. Nesterov: *Introductory lectures on convex optimization: a basic course," Kluwer Academic, 2004. $\endgroup$ Commented Nov 6, 2016 at 14:21
  • $\begingroup$ Ouch. That hurts my book budget. But, Thanks! $\endgroup$ Commented Nov 6, 2016 at 17:12
4
$\begingroup$

You can take a look at my paper at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2364989. I use a mapping from tensor to matrix that enables to write a fifth order Taylor series in matrix notation. Take for example the third order term. If you have a tensor $f_{xxx}$ of dimensions $n\times n\times n\times n$ whose $m,i,j,k$ element is the derivative of the $m$'th entry of $f$ wrt the $i,j,k$ entries of $x$, you can use the reshape function to reshape it into a matrix of dimensions $n\times n^3$. Then, $\frac{1}{6}g_{xxx}\left( x\otimes x\otimes x\right)$ is the third order term of the Taylor series in matrix notation.

$\endgroup$
2
$\begingroup$

I actually found a way to express it in matrix-vector notation.

Let $\mathbf{x}\in \mathbf{R}^n$ then the third order term you're looking for is $$\frac{1}{3!}\mathbf{x^T(x^T\mathfrak{D})x},$$

where $\mathfrak{D}$ is a $n\times n\times n$ tensor which looks like a $n\times 1$ vector with $n\times n$ matrices as its entries and it corresponds to the $n^3$ possible combinations of $3^{\text{rd}}$ order partial derivatives. Note that for functions that satisfy Schwarz' theorem, the following is true and thus the $n\times n$ matrices are symmetric: $(\partial_{ij})_k=(\partial_{ji})_k$ and $(\partial_i)_{jk}=(\partial_i)_{kj}$.

I also used the convention where the product between vector and tensor, $\mathbf{x^T\mathfrak{D}}$, should yield $\sum_i\mathfrak{D_i}x^i$, $\mathfrak{D_i}$ being every one of those $n\times n$ matrices.

$\mathbf{EDIT}$:

I actually found an easier notation:

$$\frac{1}{3!}\mathbf{x^{T}}\mathbf{\mathfrak{\tilde D}x}=\frac{1}{3!} \begin{pmatrix} x^1 & \dots & x^n \end{pmatrix} \begin{pmatrix} \mathbf x\cdot\nabla f_{11} & \dots & \mathbf x\cdot\nabla f_{1n}\\ \vdots & \ddots & \vdots\\ \mathbf x\cdot\nabla f_{n1} & \dots & \mathbf x\cdot\nabla f_{nn} \end{pmatrix} \begin{pmatrix} x^1\\ \vdots\\ x^n \end{pmatrix},$$ where the indices on the $f$s mean the partial derivatives wrt the $i$th or the $j$th coordinate.

$\endgroup$
1
  • $\begingroup$ Interestingly enough, this without the coefficient and for any three vectors, it is also the general structure for trilinear forms in analogy with bilinear form's $\mathbf{u^T Gv}$ $\endgroup$
    – Conreu
    Commented Jul 20, 2023 at 23:59
0
$\begingroup$

I have tried a bunch of approaches:

(1) Using gradient-vector products: \begin{align*} v(x + \varepsilon) &\approx v(x) + \left[\frac{\partial}{\partial x} v(x)\right]\varepsilon + \frac{1}{2}\left[\frac{\partial}{\partial x}\left[\frac{\partial}{\partial x} v(x)\right]\varepsilon\right]\varepsilon + \frac{1}{6}\left[\frac{\partial}{\partial x}\left[\frac{\partial}{\partial x} \left[\frac{\partial}{\partial x} v(x)\right]\varepsilon\right]\varepsilon\right]\varepsilon + \dots \end{align*} (2) Using the vec function, which takes a matrix $\mathbb R^{n\times m} \to \mathbb R^{nm}$: \begin{align*} v(x + \varepsilon) &\approx v(x) + \left[\frac{\partial}{\partial x} v(x)\right]\varepsilon + \frac{1}{2} \varepsilon^T \left[\frac{\partial^2 v(x)}{\partial x^2} \right]\varepsilon + \frac{1}{6} (\varepsilon \otimes \varepsilon)^T \left[\frac{\partial}{\partial x}\mathrm{vec}\!\left(\frac{\partial^2 v(x)}{\partial x^2}\right)\right]\varepsilon + \dots \end{align*} (3) Using indices: \begin{align*} v(x + \varepsilon) &\approx v(x) + \sum_j \frac{\partial v(x)}{\partial x_j} \varepsilon_j + \frac12 \sum_{j,k} \frac{\partial v(x)}{\partial x_j\partial x_k} \varepsilon_j \varepsilon_k + \frac16 \sum_{j,k,\ell} \frac{\partial v(x)}{\partial x_j\partial x_k\partial x_\ell} \varepsilon_j \varepsilon_k \varepsilon_\ell \end{align*}

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .