0
$\begingroup$

I’m struggling with how to look at the differential of a multi linear map. The setting: Let $A: \mathbb{R}^n \times … \times \mathbb{R}^n \to \mathbb{R}$ be a multi linear map with k entries. I’m asked to compute the differential of $A(x,…,x)$. I’ve tried to look at the case where $n = 1$ and $k = 2$ (and $n=2, k=2$) to look at the small cases and I think I have a general feeling of what happens here, but I’m unsure of what I think happens is actually true…

What I’ve tried: (n=1, k=2)

$A: \mathbb{R}\times\mathbb{R} \to \mathbb{R}$ then sends an element $(x,y)\mapsto A(x,y)$, the derivative is then $dA = (\partial_xA(x,y), \partial_yA(x,y))$ where $\partial_x$ is the partial derivative of $A(x,y)$ to $x$.

When n=2 (or higher), this would result in a matrix, where on the columns the derivative to that component are? But I’m not sure if this is correct…

$\endgroup$

1 Answer 1

0
$\begingroup$

$ \newcommand\diff{\mathrm D} \newcommand\R{\mathbb R} \newcommand\DD[2]{\frac{\mathrm d#1}{\mathrm d#2}} $

In this case, it is only confusing to try to think of the differential as a matrix. Instead, think of it as a linear transformation. If $f : V \to W$ for any (normed) vector spaces $V, W$, the differential $\diff f_x$ at $x \in V$ is the "best linear approximation" to $f$; that is $\diff f_x : V \to W$ is a linear function. When $V = \R^m$ and $W = \R^n$, you can represent $\diff f_x$ as a real $m\times n$ matrix just like any linear transformation. But in this case $V = \R^n\times\cdots\times\R^n$ and $W = \R$, and viewing $\diff f_x$ as a matrix is possible but not very natural.

Let's consider a function $A : V\times V \to W$; hopefully it will be clear how to generalize to an arbitrary number of parameters. The function $x \mapsto A(x, x)$ can be represented as the composition $A\circ\Delta$ where $\Delta(x) = (x, x)$. Hence by the chain rule $$ \diff[A\circ\Delta]_x = \diff A_{\Delta(x)}\circ\diff\Delta_x. $$ However, $\Delta : V \to V\times V$ is a linear function, and the differential of a linear function at any point is itself; so now we have $$ \diff[A\circ\Delta]_x = \diff A_{\Delta(x)}\circ\Delta. $$ We can write $\Delta = \delta_1 + \delta_2$ where $\delta_1(x) = (x, 0)$ and $\delta _2(x) = (0, x)$, then apply the linearity of $\diff A_{\Delta(x)}$: $$ \diff[A\circ\Delta]_x = \diff A_{\Delta(x)}\circ(\delta_1 + \delta_2) = \diff A_{\Delta(x)}\circ\delta_1 + \diff A_{\Delta(x)}\circ\delta_2. $$ Applying this on $v \in V$ yields $$ \diff[A\circ\Delta]_x(v) = \diff A_{\Delta(x)}(\delta_1(v)) + \diff A_{\Delta(x)}(\delta_2(v)). $$ Now define $A^1_x(y) = A(y,x)$ and $A^2_x(y) = A(x,y)$. The quantity $\diff A_{\Delta(x)}(\delta_1(v))$ is the directional derivative of $A$ at $(x,x)$ along $\delta_1(v)$; hence $$\begin{aligned} \diff A_{\Delta(x)}(\delta_1(v)) &= \lim_{\epsilon\to0}\frac{A(\Delta(x) + \epsilon\delta_1(v)) - A(\Delta(x))}\epsilon \\ &= \lim_{\epsilon\to0}\frac{A(x+\epsilon v, x) - A(x,x)}\epsilon \\ &= \diff[A^1_x]_x(v), \end{aligned}$$ and similarly $\diff A_{\Delta(x)}(\delta_2(v)) = \diff[A^2_x]_x(v)$. It follows that $$ \diff[A\circ\Delta]_x = \diff[A^1_x]_x + \diff[A^2_x]_x. $$ What this is saying is that to compute $\diff[A\circ\Delta]_x$ it suffices to differentiate with respect to each "slot" of $A$ separately and then add the results together. I prefer to write this in much more suggestive notation: $$ \diff[A(x,x)] = \dot\diff[A(\dot x, x)] + \dot\diff[A(x,\dot x)], $$ where the dots should be thought of as specifying what is being differentiated. The undotted $x$ should be thought of as being kept constant. If we write $\diff_y$ to specify differentiating with respect to $y$, then another way to write the same thing would be $$ \diff[A(x,x)] = \Bigl[\diff_y[A(y,x)] + \diff_y[A(x,y)]\Bigr]_{y=x}. $$

When $A$ is multilinear, it is of course linear in each argument; but as stated a linear function is its own differential, so $$ \diff[A(x,x)](v) = \dot\diff[A(\dot x,x)](v) + \dot\diff[A(x,\dot x)](v) = A(v,x) + A(x,v). $$


This idea is very powerful, and seems to not be as well-known as it should. For example, this directly implies the product rule: if $f : \R \to \R$ and $g : \R \to \R$, then $$\begin{aligned} \DD{}x[f(x)g(x)] &= \DD{}{\dot x}f(\dot x)g(x) + \DD{}{\dot x}f(x)g(\dot x) \\ &= \DD fxg(x) + f(x)\DD gx. \end{aligned}$$ It also extends to gradients since $(v\cdot\nabla)f(x) = \diff f_x(v)$. If $f, g : \R^n \to \R$ then $$ \nabla(f(x)g(x)) = \dot\nabla f(\dot x)g(x) + \dot\nabla f(x)g(\dot x) = (\nabla f(x))g(x) + f(x)(\nabla g(x)). $$ If $f, g : \R^n \to \R^m$ then $$ \nabla(f(x)\cdot g(x)) = \dot\nabla(f(\dot x)\cdot g(x)) + \dot\nabla(f(x)\cdot g(\dot x)), $$ which is something that cannot be expressed well without overdots. When $f = g$ this tells us that $$ \nabla(f(x)\cdot f(x)) = 2\dot\nabla(f(\dot x)\cdot f(x)). $$ More interesting examples come when we consider non-linear functions. For example $$ \DD{}x x^x = \DD{}{\dot x}\dot x^x + \DD{}{\dot x}x^{\dot x} = xx^{x-1} + x^x\ln x = x^x(1 + \ln x). $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.