10
$\begingroup$

What is the third order term in the Taylor Series Expansion? I know it will just be third order partial derivatives but I want to know how is it expressed in a compact Matrix notation. For instance Jacobian for first order, Hessian for second order partial derivatives.

In other words, what is the third order term in the equation below? Thanks for your help!

Second Order Taylor Series Expansion

$\endgroup$
  • $\begingroup$ the third order term is connected with a trilinear, completely symmetric mapping. I'm not sure I've ever seen this written as a matrix multiplication. Fortunately, linear and quadratic forms permit a natural matrix representation, but, well, I await answers... How will we get a trilinear mapping on $\mathbb{R}^n$ from a multiplication with an $n \times n$ matrix? $\endgroup$ – James S. Cook Nov 8 '13 at 15:19
  • $\begingroup$ Matrix notation would make it really easy to understand and program. If that is not possible, then is there an intuitive way to understand what all terms will be of the third order and what their coefficients will be? $\endgroup$ – Innocent Nov 8 '13 at 15:27
7
$\begingroup$

Well, $f: \mathbb{R}^n \rightarrow \mathbb{R}^n$ has multivariate Taylor expansion at $x+h$ centered at $x$ of: $$ f(x+h) = f(x)+\sum_{i=1}^n \frac{\partial f}{\partial x_i}h_i +\frac{1}{2}\sum_{i,j=1}^n \frac{\partial^2 f}{\partial x_i\partial x_j}h_ih_j+\frac{1}{6}\sum_{i,j,k=1}^n \frac{\partial^3 f}{\partial x_i\partial x_j\partial x_k}h_ih_jh_k+ \cdots$$ Here we could define $T: \mathbb{R}^n \times \mathbb{R}^n \times \mathbb{R}^n \rightarrow \mathbb{R}^n $ by its values on the basis elements $e_i,e_j,e_k$ (unit-vector basis for Cartesian coordinates in $n$-dimensions) $$ T_{ijk}=\sum_{i,j,k=1}^n \frac{\partial^3 f}{\partial x_i\partial x_j\partial x_k} $$ The question is, can we write the formula $\sum_{i,j,k=1}^nT_{ijk}h_ih_jh_k$ as a matrix multiplication of some sort... I suggest we consider $\mathbb{R}$-valued $f$ as a starting point.

(with the proper conventions set forth there is a way to write this as a matrix multiplication, but, I think in the process of doing such we lose track of the manifest trilinearity, I'm not well-versed in the needed notation at this point in time so I'll leave my answer here as it stands since expressing this as a matrix multiplication on an abstract space is not that interesting, well, at least to me today)

$\endgroup$
  • $\begingroup$ Is there a name for $T_{ijk}$? $\endgroup$ – jvriesem Dec 11 '15 at 17:41
  • 2
    $\begingroup$ @jvriesem It's the Tressian. No, just kidding, I know the second order piece is commonly referred to as the Hessian. I've not seen an in-depth analysis of just the third order piece. Anyway, I am not aware of a name. $\endgroup$ – James S. Cook Dec 12 '15 at 3:22
3
$\begingroup$

$\renewcommand{\Re}{\mathbb{R}}$Often Taylor expansions of functions $f:\Re^n\to\Re$ at a point $x$ are meant along a given direction $d\in\Re^n$. This facilitates a lot out understanding even for first-order expansions.

Let $f\in\mathcal{C}^3$ and define a function $\phi:\Re\to\Re$ given by $\phi(\tau) = f(x+\tau d)$; this is like a slice of $f$: it describes $f$ along the line $x+\tau d,\tau\in\Re$. Then $\phi$ is three times continuously differentiable and the third order expansion of $\phi$ about $\tau=0$ is

$$\begin{aligned} \phi(\tau) = \phi(0) + \tau \phi'(0) + \tfrac{t^2}{2!}\phi''(0) + \tfrac{t^3}{3!}\phi'''(0) + o(t^3). \end{aligned}$$

But $\phi'(0)$ is related to the directional derivative of $f$ at $x$ along the direction $d$ which is

$$\begin{aligned} \phi'(0) &= \lim_{h\to 0}\frac{\phi(h) - \phi(0)}{h}\\ &= \lim_{h\to 0}\frac{f(x+h d) - f(x)}{h}\\ &= \langle \nabla f(x), d \rangle \end{aligned}$$

Let us denote this by $\nabla_{d}f(x)$.

Similary, $\phi''(0)$ can be interpreted as the directional Hessian of $f$ at $x$ along the directions $d$ and $d$, that is

$$\begin{aligned} \phi''(0) = \langle \nabla^2 f(x)d, d \rangle \end{aligned}$$

Let us denote this by $\nabla^2_{d,d}f(x)$.

The term $\phi'''(0)$ - the "Tressian" as James S. Cook jokingly put it - is more difficult to represent. Indeed, it will be a tensor as Oren explained. However, we are merely interested in the "directional Tressian" of $f$ at $x$ along directions $d$, $d$ and $d$. This construct is actually used in the context of convex optimization theory and in particular the theory of self-concordant functions and is denoted by $\nabla^3_{d,d,d}f(x)$ and we may write

$$\begin{aligned} \nabla^3_{d,d,d}f(x) = \langle \nabla^3 f(x)[d]d, d \rangle \end{aligned}$$

where $\nabla^3 f(x)$ is the third-order gradient of $f$ at $x$ which, in my opinion, is best understood via its directional variant:

$$\begin{aligned} \nabla^3f(x)[d] = \lim_{h\to 0} \frac{\nabla^2 f(x+\alpha d) - \nabla^2 f(x)}{h} \end{aligned}$$

Here $\nabla^3f(x)[d]$ is a matrix - it is a directional Hessian. Essentially, $\nabla^3f(x)[d]$ describes how the Hessian of $f$ changes at $x$ along the direction $d$.

$\endgroup$
  • $\begingroup$ Interesting. Where can we read more about this? $\endgroup$ – James S. Cook Nov 5 '16 at 23:03
  • $\begingroup$ @JamesS.Cook I think a good reference is the book of Y. Nesterov: *Introductory lectures on convex optimization: a basic course," Kluwer Academic, 2004. $\endgroup$ – Pantelis Sopasakis Nov 6 '16 at 14:21
  • $\begingroup$ Ouch. That hurts my book budget. But, Thanks! $\endgroup$ – James S. Cook Nov 6 '16 at 17:12
2
$\begingroup$

You can take a look at my paper at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2364989. I use a mapping from tensor to matrix that enables to write a fifth order Taylor series in matrix notation. Take for example the third order term. If you have a tensor $f_{xxx}$ of dimensions $n\times n\times n\times n$ whose $m,i,j,k$ element is the derivative of the $m$'th entry of $f$ wrt the $i,j,k$ entries of $x$, you can use the reshape function to reshape it into a matrix of dimensions $n\times n^3$. Then, $\frac{1}{6}g_{xxx}\left( x\otimes x\otimes x\right)$ is the third order term of the Taylor series in matrix notation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.