# partitioned matrix differentiation

I know how to differentiate a product of matrices wrt a given matrix-entry e.g., $$X_{i, j}$$. However, I'm not sure if I'm thinking correctly about how to do that when I have a vector i.e., just one column.

I have $$\mathbf{x^{\top}Ax}$$ and I want it's derivative wrt $$x_{i}$$. Suppose that $$\mathbf{x}$$ is $$4 \times 1$$ and that $$\mathbf{A}$$ is $$4 \times 4$$ and symmetric! What follows is correct?

$$\frac{\partial \mathbf{x^{\top}Ax}}{\partial x_{1}} = 2\mathbf{J}^{1\cdot}\mathbf{Ax} = 2 \begin{bmatrix} 1 & 1 & 1 & 1\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 \end{bmatrix} \mathbf{Ax}.$$

• how did you compute that expression? – Exodd Apr 7 at 20:26
• Why is your partial derivative a $4 \times 1$ vector? $\frac{\partial x^TAx}{\partial x_1}$ should be a scalar – Ben Grossmann Apr 7 at 20:27
• So the answer is no: your answer is not correct. – Ben Grossmann Apr 7 at 20:28
• We know that $\partial \mathbf{x^{\top}Ax}/\partial \mathbf{x} = 2\mathbf{Ax}$. When we deal with $\mathbf{X}$ instead of $\mathbf{x}$ and differentiate wrt $X_{ij}$, we introduce the single-entry matrix $\mathbf{J}^{ij}$. What I did here was to 'adapt' the idea of the single-entry matrix to the context of a vector $\mathbf{x}$. @Omnomnomnom, you're saying that since I'm differentiating in one value, my output should be a scalar, not a vector - even with the other elements of that vector being zero (as I proposed), right? What you said makes sense to me, but then I don't know how to solve it. – Henrique Laureano Apr 7 at 20:56
• Okay, you know the gradient is $2Ax$, so if you multiply by the $k^{th}$ basis vector you'll get $$2e_k^TAx = e_k^T\left(\frac{\partial f}{\partial x}\right) = \frac{\partial f}{\partial x_k}$$ Think of $e_k$ as the single-entry vector. – greg Apr 7 at 21:13

The answer is simple, Greg explained clearly in the comments section.

We know that $$\frac{\partial \mathbf{x}^{\top}\mathbf{Ax}}{\partial \mathbf{x}} = 2\mathbf{Ax}.$$ So, if now I want the derivative wrt $$x_{k}$$ I just multiply it by the $$k^{\text{th}}$$ basis vector. i.e., $$\frac{\partial \mathbf{x}^{\top}\mathbf{Ax}}{\partial x_{k}} = 2e_{k}^{\top}\mathbf{Ax} = e_{k}^{\top} \frac{\partial \mathbf{x}^{\top}\mathbf{Ax}}{\partial \mathbf{x}}.$$

I assume that $$A$$ is a matrix having real numbers as entries. Hence with $$x^T:=(x_1,\ldots,x_n)$$ the term $$x^TAx$$ is a quadratic polynomial in $$n$$ variables $$x_1,\ldots,x_n$$ which you can differentiate w.r.t. $$x_1$$.

To your calculation: That equation does not hold, since $$x^TAx$$ is a real number for all $$x_1,\ldots,x_n$$ and not $$4\times 1$$ as in the middle and rhs.

• If I have $\mathbf{x}^{\top} = [x_{1} x_{2}]$ and $\mathbf{A}$ with entries, $a, b, c, d$, $\mathbf{x}^{\top}\mathbf{Ax} = ax_{1}^{2} + (b+c)x_{1}x_{2} + dx_{2}^{2}$. Are you saying that the derivative of this polynomial wrt $x_{1}$ is my answer? Ok, but then how I generalize it in a matrix form? – Henrique Laureano Apr 7 at 21:09
• If $A$ is a $n\times n$ matrix and also $x$ is a $n\times k$ matrix with monomials as entries then you can compute the matrix product and derivate the result entrywise. Since the set of all $l\times k$ matrices $Mat(l,k;\mathbb{R})$ is a vector space you are able to define a notion of differentiability for maps $A: \mathbb{R}\longrightarrow Mat(l,k;\mathbb{R})$, which you would have in that case. And yes, that was what I proposed. Just the partial differential of $ax^2_1+(b+c)x_1x_2+dx_2^2$ – dennis_s Apr 8 at 10:45