Product rule for matrix-valued functions and Differentiability of matrix multiplication

Let $$F, G: \mathbb{R}^{n \text{ x } n} \to \mathbb{R}^{n \text{ x } n}$$ be two differentiable functions. Defining $$(F \cdot\ G)(A):=F(A)G(A)$$ using the usual matrix multiplication, prove that $$F\cdot\ G$$ is differentiable and express $$(F\cdot\ G)'$$ in terms of $$F$$, $$F'$$, $$G$$, $$G'$$.

What I thought of doing was to write $$A=[a_{ij}]_{i, j \in {1, ..., n}}$$ and express $$F$$ as a matrix $$[f_{ij}]_{i, j \in {1, ..., n}}$$ of differentiable functions $$f_{ij}:\mathbb{R} \to \mathbb{R}$$ (same idea for $$G$$), and I imagine $$(F\cdot\ G)'$$ will look something like $$F'\cdot\ G+F\cdot\ G'$$, but I don't know how to formalize it.

[obs.: the matrix norm considered here is $$||A||=\sup_{|v|=1}|A(v)|$$, where $$v \in \mathbb{R^n}$$ and $$|\cdot\ |$$ is the usual euclidean norm]

• You'll get some up-votes if you mention how you tried to solve this upto some point atleast :) – Blogger Jun 9 '16 at 1:43
• May I ask what is $F(T)$? I thought $T$ was the input but it seems it is a linear transformation? – Vim Jun 9 '16 at 2:53
• What type of norm are you using on $\mathcal L(\mathbb R^n)$? Do you want the usual operator norm and to consider bounded linear operators? Also, writing $F = [f_{ij}]$ assumes $F$ is a linear map. – Jon Warneke Jun 9 '16 at 3:01
• edited for clarification – rmdmc89 Jun 9 '16 at 3:42
• So this is basically asking if composition of linear operators in $\mathcal{L}(\mathbb{R}^n)$ is a differentiability preserving map? If that's the case you could probably prove it pretty easily by breaking $F$ and $G$ down into their element-wise action on the entries of $A$ and showing that $F(A)G(A)$ is just a bunch of addition and multiplication, which you already know preserve differentiability because the field is $\mathbb{R}$. At that point expressing $(F(A)G(A))'$ would just be an application of the power rule....correct me if I'm wrong on any of this. – John Cramerus Jun 9 '16 at 3:58

The guess was more or less in the right direction, but the meaning of $$F'G+FG'$$ needs to be stated precisely.

First, we should clarify some notation: for $$M_0 \in \mathbb{R}^{n\times n}$$, the expression $$F'(M_0)[M]$$ means the derivative of $$F$$ at $$M_0$$, applied to $$M$$, for $$M\in \mathbb{R}^{n\times n}$$. Letting $$M_0$$ be fixed, we'll also use the notation $$\Delta F(M):=F(M_0+M)-F(M_0)$$.

Now what we will prove is this:

$$(F\cdot\ G)'(M_0)[M]=F'(M_0)[M]G(M_0)+F(M_0)G'(M_0)[M] \text{, for all }M\in\mathbb{R}^{n\times n}$$

(notice that $$M$$ - and every other matrix for that matter - needs to be in the right places, since matrix multiplication is not commutative)

By definition, we need to show that:

$$\lim_{||M||\to 0}\frac{||\Delta (F\cdot\ G)(M)-(F'(M_0)[M]G(M_0)+F(M_0)G'(M_0)[M])||}{||M||}=0 \text{ (*)}$$

Let's make some rearrangements. Notice that:

\begin{align*} \Delta (F\cdot\ G)(M) &=F(M_0+M)G(M_0+M)-F(M_0)G(M_0) \\ &=(\Delta F(M)+F(M_0))(\Delta G(M)+G(M_0))-F(M_0)G(M_0)\\ &=\Delta F(M) \Delta G[M] + \Delta F(M) G(M_0)+F(M_0)\Delta G(M) \end{align*}

Pluging this in (*) and rearranging the terms, what we actually need to show is:

$$\frac{||\Delta F(M)\Delta G(M)+(\Delta F(M)-F'(M_0)[M])G(M_0)+F(M_0)(\Delta G(M)-G'(M_0)[M])||}{||M||}\to 0$$

as $$||M||\to 0$$. Using the triangle inequality and the fact that $$||A\cdot\ B||\leq ||A||\cdot\ ||B||$$ we have that the last expression is less or equal to the sum:

$$\frac{||\Delta F(M)\Delta G(M)||}{||M||}+ \frac{||\Delta F(M)-F'(M_0)[M]||}{||M||}||G(M_0)||+||F(M_0)||\frac{||\Delta G(M)-G'(M_0)[M]||}{||M||}$$

Notice that since $$F, G$$ are differentiable, the last two terms go to zero, by definition of derivative. So we're only left to prove that the first term $$\frac{||\Delta F(M)\Delta G(M)||}{||M||}$$ goes to zero.

Since $$\Delta F(M) = F'(M_0)[M] + o(||M||)$$ and $$\Delta G(M) = G'(M_0)[M] + o(||M||)$$, we get:

$$\Delta F(M_0)\Delta G(M_0)= F'(M_0)[M]G'(M_0)[M]+o(||M||)$$

as $$||M||\to 0$$. Now, since $$M$$, $$F'(M_0)$$ and $$G'(M_0)$$ are linear operators themselves, we have:

\begin{align*} \lim_{||M||\to 0}\frac{||\Delta F(M_0)\Delta G(M_0)||}{||M||} &= \lim_{||M||\to 0}\frac{||F'(M_0)[M]G'(M_0)[M]||}{||M||} \\ &\leq \lim_{||M||\to 0}{\frac{||F'(M_0)[M]||}{||M||}||G'(M_0)[M]||}\\ &\leq \lim_{||M||\to 0}\frac{||F'(M_0)||\cdot\ ||M||}{||M||} ||G'(M_0)[M]||\\ &= ||F'(M_0)||\lim_{||M||\to 0}||G'(M_0)[M]|| \end{align*}

But linear operators are continuous, so $$||G'(M_0)[M]||\to 0$$ as $$||M||\to 0$$ and we're done.

$$E = \mathbb{R}^{n \times n}$$ is a Banach space (which, with the given norm, is isomorphic to the space of continuous linear maps $$\mathscr{L}(\mathbb{R}^n; \mathbb{R}^n)$$).

The usual matrix multiplication $$\mu \colon E \times E \to E$$, $$(B, C) \mapsto BC$$ is bilinear; and it is continuous, because $$\|BC\| \leqslant \|B\|\|C\|$$ for all $$B, C \in E$$. Continuity follows from e.g. J. Dieudonne, Foundations of Modern Analysis (1969), proposition (5.5.1).

By e.g. Dieudonne's proposition (8.1.4), $$\mu$$ is differentiable at every point $$(B, C)$$ in $$E \times E$$, and its derivative at $$(B, C)$$ is the continuous linear mapping $$$$\tag{1}\label{eq:mu} \mu'(B, C) \colon E \times E \to E, \ (U, V) \mapsto BV + UC.$$$$

By e.g. Dieudonne's proposition (8.1.5), the function $$H \colon E \to E \times E, \ A \mapsto (F(A), G(A))$$ is differentiable at every point $$A \in E$$, and - under the natural linear isometry between $$\mathscr{L}(E; E \times E)$$ and $$\mathscr{L}(E; E) \times \mathscr{L}(E; E)$$, where both products have the $$\sup$$ norm - the derivative at $$A$$ is identified with the continuous linear mapping $$H'(A) = (F'(A), G'(A)).$$

(I had better now spell that out in more detail - especially as I haven't properly studied differential calculus in Banach spaces! It took some brass nerve to post this answer. Partly it's in order to learn some of this stuff myself, but it is also to show that an answer requires virtually no calculation.)

The natural linear isometry in question sends $$L : E \to E \times E$$ to $$(\operatorname{pr}_1 \circ L, \operatorname{pr}_2 \circ L)$$, where $$\operatorname{pr}_i$$ ($$i = 1, 2$$) are the projection mappings $$E \times E \to E$$. So we have $$$$\tag{2}\label{eq:H} H'(A) \colon E \to E \times E, \ T \mapsto (F'(A)(T), G'(A)(T)).$$$$

By the Chain Rule (which is Dieudonne's proposition (8.1.2)), \begin{align*} (F \cdot G)'(A) & = (\mu \circ H)'(A) \\ & = \mu'(H(A)) \circ H'(A) \\ & = \mu'(F(A), G(A)) \circ H'(A). \end{align*} Therefore, by \eqref{eq:mu} and \eqref{eq:H}, \begin{align*} (F \cdot G)'(A)(T) & = \mu'(F(A), G(A))(H'(A)(T)) \\ & = \mu'(F(A), G(A))(F'(A)(T), G'(A)(T)) \\ & = \boxed{F(A)G'(A)(T) + F'(A)(T)G(A)} \end{align*} - which agrees with rmdmc89's answer.

By the same argument, the result holds generally for differentiable functions $$F: W \to \mathbb{R}^{n \times p}$$, $$G: W \to \mathbb{R}^{m \times n}$$, where $$W$$ is an open subset of a Banach space and $$m, n, p$$ are positive integers.

• If it would help to make the answer more self-contained, and to show that Dieudonne's bible is worth consulting on the question, I could copy out chapter and verse of some of the most relevant propositions - but perhaps that would be of as little use to those who don't have The Book to hand as to those who do! (On the plus side, however, it would give me an excuse to type the words "slavish subservience to the shibboleth of numerical interpretation at any cost"!) :) – Calum Gilhooley Nov 15 '18 at 18:54