# Differential of Multi Linear Map

I’m struggling with how to look at the differential of a multi linear map. The setting: Let $$A: \mathbb{R}^n \times … \times \mathbb{R}^n \to \mathbb{R}$$ be a multi linear map with k entries. I’m asked to compute the differential of $$A(x,…,x)$$. I’ve tried to look at the case where $$n = 1$$ and $$k = 2$$ (and $$n=2, k=2$$) to look at the small cases and I think I have a general feeling of what happens here, but I’m unsure of what I think happens is actually true…

What I’ve tried: (n=1, k=2)

$$A: \mathbb{R}\times\mathbb{R} \to \mathbb{R}$$ then sends an element $$(x,y)\mapsto A(x,y)$$, the derivative is then $$dA = (\partial_xA(x,y), \partial_yA(x,y))$$ where $$\partial_x$$ is the partial derivative of $$A(x,y)$$ to $$x$$.

When n=2 (or higher), this would result in a matrix, where on the columns the derivative to that component are? But I’m not sure if this is correct…

$$\newcommand\diff{\mathrm D} \newcommand\R{\mathbb R} \newcommand\DD{\frac{\mathrm d#1}{\mathrm d#2}}$$
In this case, it is only confusing to try to think of the differential as a matrix. Instead, think of it as a linear transformation. If $$f : V \to W$$ for any (normed) vector spaces $$V, W$$, the differential $$\diff f_x$$ at $$x \in V$$ is the "best linear approximation" to $$f$$; that is $$\diff f_x : V \to W$$ is a linear function. When $$V = \R^m$$ and $$W = \R^n$$, you can represent $$\diff f_x$$ as a real $$m\times n$$ matrix just like any linear transformation. But in this case $$V = \R^n\times\cdots\times\R^n$$ and $$W = \R$$, and viewing $$\diff f_x$$ as a matrix is possible but not very natural.
Let's consider a function $$A : V\times V \to W$$; hopefully it will be clear how to generalize to an arbitrary number of parameters. The function $$x \mapsto A(x, x)$$ can be represented as the composition $$A\circ\Delta$$ where $$\Delta(x) = (x, x)$$. Hence by the chain rule $$\diff[A\circ\Delta]_x = \diff A_{\Delta(x)}\circ\diff\Delta_x.$$ However, $$\Delta : V \to V\times V$$ is a linear function, and the differential of a linear function at any point is itself; so now we have $$\diff[A\circ\Delta]_x = \diff A_{\Delta(x)}\circ\Delta.$$ We can write $$\Delta = \delta_1 + \delta_2$$ where $$\delta_1(x) = (x, 0)$$ and $$\delta _2(x) = (0, x)$$, then apply the linearity of $$\diff A_{\Delta(x)}$$: $$\diff[A\circ\Delta]_x = \diff A_{\Delta(x)}\circ(\delta_1 + \delta_2) = \diff A_{\Delta(x)}\circ\delta_1 + \diff A_{\Delta(x)}\circ\delta_2.$$ Applying this on $$v \in V$$ yields $$\diff[A\circ\Delta]_x(v) = \diff A_{\Delta(x)}(\delta_1(v)) + \diff A_{\Delta(x)}(\delta_2(v)).$$ Now define $$A^1_x(y) = A(y,x)$$ and $$A^2_x(y) = A(x,y)$$. The quantity $$\diff A_{\Delta(x)}(\delta_1(v))$$ is the directional derivative of $$A$$ at $$(x,x)$$ along $$\delta_1(v)$$; hence \begin{aligned} \diff A_{\Delta(x)}(\delta_1(v)) &= \lim_{\epsilon\to0}\frac{A(\Delta(x) + \epsilon\delta_1(v)) - A(\Delta(x))}\epsilon \\ &= \lim_{\epsilon\to0}\frac{A(x+\epsilon v, x) - A(x,x)}\epsilon \\ &= \diff[A^1_x]_x(v), \end{aligned} and similarly $$\diff A_{\Delta(x)}(\delta_2(v)) = \diff[A^2_x]_x(v)$$. It follows that $$\diff[A\circ\Delta]_x = \diff[A^1_x]_x + \diff[A^2_x]_x.$$ What this is saying is that to compute $$\diff[A\circ\Delta]_x$$ it suffices to differentiate with respect to each "slot" of $$A$$ separately and then add the results together. I prefer to write this in much more suggestive notation: $$\diff[A(x,x)] = \dot\diff[A(\dot x, x)] + \dot\diff[A(x,\dot x)],$$ where the dots should be thought of as specifying what is being differentiated. The undotted $$x$$ should be thought of as being kept constant. If we write $$\diff_y$$ to specify differentiating with respect to $$y$$, then another way to write the same thing would be $$\diff[A(x,x)] = \Bigl[\diff_y[A(y,x)] + \diff_y[A(x,y)]\Bigr]_{y=x}.$$
When $$A$$ is multilinear, it is of course linear in each argument; but as stated a linear function is its own differential, so $$\diff[A(x,x)](v) = \dot\diff[A(\dot x,x)](v) + \dot\diff[A(x,\dot x)](v) = A(v,x) + A(x,v).$$
This idea is very powerful, and seems to not be as well-known as it should. For example, this directly implies the product rule: if $$f : \R \to \R$$ and $$g : \R \to \R$$, then \begin{aligned} \DD{}x[f(x)g(x)] &= \DD{}{\dot x}f(\dot x)g(x) + \DD{}{\dot x}f(x)g(\dot x) \\ &= \DD fxg(x) + f(x)\DD gx. \end{aligned} It also extends to gradients since $$(v\cdot\nabla)f(x) = \diff f_x(v)$$. If $$f, g : \R^n \to \R$$ then $$\nabla(f(x)g(x)) = \dot\nabla f(\dot x)g(x) + \dot\nabla f(x)g(\dot x) = (\nabla f(x))g(x) + f(x)(\nabla g(x)).$$ If $$f, g : \R^n \to \R^m$$ then $$\nabla(f(x)\cdot g(x)) = \dot\nabla(f(\dot x)\cdot g(x)) + \dot\nabla(f(x)\cdot g(\dot x)),$$ which is something that cannot be expressed well without overdots. When $$f = g$$ this tells us that $$\nabla(f(x)\cdot f(x)) = 2\dot\nabla(f(\dot x)\cdot f(x)).$$ More interesting examples come when we consider non-linear functions. For example $$\DD{}x x^x = \DD{}{\dot x}\dot x^x + \DD{}{\dot x}x^{\dot x} = xx^{x-1} + x^x\ln x = x^x(1 + \ln x).$$