# Confusion with matrix derivatives and component-wise gradients

According to this source (The Matrix Cookbook), we have that for $$f(X) = \mathbf{a}^T X \mathbf{b}$$ where $$X \in \mathbb{R}^{n \times m}, \mathbf{a} \in \mathbb{R}^n, \mathbf{b} \in \mathbb{R}^m$$, the derivative is $$\frac{\partial f}{\partial X} = \mathbf{a} \mathbf{b}^T$$ which is a scalar since it's basically a dot product. I see how we can arrive at this result by applying matrix derivative rules/applying basic derivative rules to vectors/matrices. However, taking the derivative (gradient) of a function $$f: \mathbb{R}^{n \times m} \to \mathbb{R}$$ based on my my calculus knowledge should yield a vector/matrix of partial derivatives.

As an example, I tried this with $$\mathbf{a} = \begin{bmatrix} a_1 & a_2 \end{bmatrix}, \mathbf{b} = \begin{bmatrix} b_1 & b_2 \end{bmatrix}$$, and $$X = \begin{bmatrix} x_1 & x_2 \\ x_3 & x_4 \end{bmatrix}$$. I expanded the expression and took the gradient to get that $$\nabla f = \begin{bmatrix} \frac{\partial f}{\partial x_1} & \frac{\partial f}{\partial x_2} \\ \frac{\partial f}{\partial x_3} & \frac{\partial f}{\partial x_4} \end{bmatrix} = \begin{bmatrix} a_1b_1 & a_1b_2 \\ a_2b_1 & a_2b_2 \end{bmatrix}.$$

So, my question is, is this the derivative, or is it $$\mathbf{a}\mathbf{b}^T$$? Should the derivative be a vector or a scalar?

P.S. I also noticed that $$\mathbf{a}\mathbf{b}^T = \operatorname{trace}(\nabla f)$$ for the matrix I computed for $$\nabla f$$. Is there any significance to that?

• It's not a scalar .
– lcv
Commented Feb 15 at 12:03

The problem is vector usually refers to a column vector (otherwise the expression $$a^TXb$$ doesn't even make sense since the dimensions don't match). If you instead do $$ab^T = \begin{pmatrix}a_1\\a_2\end{pmatrix}\begin{pmatrix}b_1 & b_2\end{pmatrix} = \begin{pmatrix} a_1b_1 & a_1b_2\\ a_2b_1 & a_2b_2 \end{pmatrix}$$ then you get the same result as your computation.

• Hmm, I see. I was not considering the outer product when I did this, but I suppose that makes sense. Commented Feb 14 at 4:44

There are several competing notations. The notation in Matrix Cookbook can be motivated as follows -- suppose we minimize scalar function of scalar variable $$w$$ using gradient descent update

$$w\leftarrow w-f'(w)$$

Now, suppose $$f(W)$$ is a scalar function of matrix variable. We want similar looking update formula to hold

$$W\leftarrow W-f'(W)$$

So this determines the shape of $$f'$$: $$i,j$$th entry of $$f'$$ is the derivative of $$f$$ w.r.t. to $$i,j$$the entry of $$W$$, which gives you $$f'(W)=ab^T$$ when $$f(W)=a'Wb$$

• Thanks for the reply. Can you explain how differentiating $f$ w.r.t the $i,j$th entry of $W$ gives $\mathbf{a}\mathbf{b}^T$? Wouldn't that yield the same matrix in my answer? Commented Feb 14 at 4:43

The Jacobian matrix is the derivative of a vector function $$\pmb{f}$$ of a vector $$\pmb{x}$$: $$\frac{\partial \pmb{f}}{\partial \pmb{x}^T} = \begin{bmatrix}\frac{\partial f_1}{\partial x_{1}} & \frac{\partial f_1}{\partial x_{2}} \\ \frac{\partial f_2}{\partial x_{1}} & \frac{\partial f_2}{\partial x_2} \end{bmatrix}$$ i.e. the partial derivatives are w.r.t. the transpose of the vector. The relation between the differential of $$\pmb{f}$$ and the differential of $$\pmb{x}$$ is, \begin{align*} d\pmb{f}= \begin{bmatrix} df_1 \\ df_2 \end{bmatrix} & = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} dx_1 + \frac{\partial f_1}{\partial x_2} dx_2 \\ \frac{\partial f_2}{\partial x_1} dx_1 + \frac{\partial f_2}{\partial x_2} dx_2 \end{bmatrix} \\ &= \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \frac{\partial f_1}{\partial x_2} \\ \frac{\partial f_2}{\partial x_1} & \frac{\partial f_2}{\partial x_2} \end{bmatrix} \begin{bmatrix} dx_1 \\ dx_2 \end{bmatrix} =\frac{\partial \pmb{f}}{\partial \pmb{x}^T} d\pmb{x} \end{align*} which shows that the definition of the derivative of $$\pmb{f}$$ w.r.t. $$\pmb{x}$$ as $$\frac{\partial \pmb{f}}{\partial \pmb{x}^T}$$ is correct.

In the same way given $$f=\pmb{a}^T\pmb{X}\pmb{b}$$ we have, $$f = \begin{bmatrix}a_1 & a_2 \end{bmatrix} \begin{bmatrix}X_{11} & X_{12} \\ X_{21} & X_{22} \end{bmatrix} \begin{bmatrix}b_1 \\ b_2 \end{bmatrix}$$ and, $$\frac{\partial f}{\partial \pmb{X}^T} = \begin{bmatrix}\frac{\partial f}{X_{11}} & \frac{\partial f}{X_{21}} \\ \frac{\partial f}{X_{12}} & \frac{\partial f}{X_{22}} \end{bmatrix}$$ i.e. the partial derivatives are w.r.t. the transpose of the matrix. Since, $$f=a_1 (b_1X_{11}+b_2X_{12}) +a_2(b_1X_{21}+b_2X_{22})$$ we have, \begin{align*} \frac{\partial f}{\partial X_{11}} &= a_1 b_1 & \frac{\partial f}{\partial X_{21}} &= a_2 b_1 \\ \frac{\partial f}{\partial X_{12}} &= a_1 b_2 & \frac{\partial f}{\partial X_{22}} &= a_2 b_2 \end{align*} and so, $$\frac{\partial f}{\partial \pmb{X}^T} = \begin{bmatrix} a_1b_1 & a_2b_1 \\ a_1b_2 & a_2b_2 \end{bmatrix} = \begin{bmatrix} b_1 \\ b_2 \end{bmatrix} \begin{bmatrix} a_1 & a_2 \end{bmatrix}= \pmb{b}\pmb{a}^T$$

The differential of $$f$$, \begin{align*} df & = \frac{\partial f}{\partial X_{11}} d X_{11} + \frac{\partial f}{\partial X_{12}} d X_{12} + \frac{\partial f}{\partial X_{21}}d X_{21} + \frac{\partial f}{\partial X_{22}}d X_{22} \\ & =a_1b_1 dX_{11} + a_1b_2 dX_{12} +a_2b_1 dX_{21} + a_2b_2 dX_{22} \\ & =\mathrm{tr} \left( \begin{bmatrix} a_1b_1 & a_2b_1 \\ a_1b_2 & a_2b_2 \end{bmatrix} \begin{bmatrix} dX_{11} & dX_{12} \\ dX_{21} & dX_{22} \end{bmatrix} \right) \\ & = \mathrm{tr} \left(\frac{\partial f}{\partial \pmb{X}^T} d\pmb{X} \right) \end{align*} and so defining the derivative of $$f$$ w.r.t. $$\pmb{X}$$ as above generates the correct differential.