Partial Derivatives involving Vectors and Matrices

Question

Let $Y$ be a $(N \times 1)$ vector, $X$ be a $N \times M$ matrix and $\lambda$ be a $M \times 1$ vector.

I am wondering how I can evaluate the following partial derivative.

\begin{align} \frac{\partial (Y-X\lambda)^T (Y-X\lambda)}{\partial \lambda_j} \end{align}

where $j = 1 \ldots P$

I run into such problems fairly often and would very much appreciate it if anyone could post a guide on differentiate expressions involving vectors and matrices (and in particular transposes / products of matrices).

Answer 1

Let's consider a more general case. Let $a$ be a vector in $\mathbb R^m$. You basically want,

$$a \cdot \nabla_\lambda [Y-\underline X(\lambda)]^2$$

where $\nabla_\lambda$ is a differential operator corresponding to derivatives with respect to $\lambda$. The chain rule tells that this is equal to

$$(a \cdot \nabla_\lambda [Y-\underline X(\lambda)] )\cdot \nabla_{Y-\underline X(\lambda)} [Y-\underline X(\lambda)]^2$$

To evaluate this, you just need a couple results about directional derivatives. First, $a \cdot \nabla_x \underline T(x) = \underline T(a)$ for any vectors $a, x$ and linear operator $\underline T$. Second, $a \cdot \nabla_x x^2 = 2a\cdot x$. With all this in mind, the expression reduces to

$$-\underline X(a) \cdot 2[Y-\underline X(\lambda)] = -2a \cdot \overline X[Y-\underline X(\lambda)]$$

To pick out a particular $\lambda_j$, simply pick $a$ such that it is full of zero entries save for a 1 in the right place.

All this follows from the chain rule and a couple basic identities for vector derivatives. Edit: Overall, it's easier (in my opinion) to consider matrix operations abstractly as linear maps instead of as matrices proper. This is basically what I've done here. Multiplication of matrices just becomes composition of functions, and that's something we know how to attack via the chain rule.

Answer 2

See the entry on Matrix Calculus in Wikipedia, or search for "matrix calculus" on the internet. In your particular case, $$ \frac{\partial (Y-X\lambda)^T (Y-X\lambda)}{\partial\lambda^T}=-2(Y-X\lambda)^T X $$ and hence the partial derivative w.r.t. $\lambda_j$ is the $j$-th entry of the row vector $-2(Y-X\lambda)^T X$.