Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Let $Y$ be a $(N \times 1)$ vector, $X$ be a $N \times M$ matrix and $\lambda$ be a $M \times 1$ vector.

I am wondering how I can evaluate the following partial derivative.

\begin{align} \frac{\partial (Y-X\lambda)^T (Y-X\lambda)}{\partial \lambda_j} \end{align}

where $j = 1 \ldots P$

I run into such problems fairly often and would very much appreciate it if anyone could post a guide on differentiate expressions involving vectors and matrices (and in particular transposes / products of matrices).

share|cite|improve this question
Do you mean $\partial (Y-X\lambda)^T(Y-X\lambda)/\partial \lambda_j$? You seem to be missing a partial in the denominator. – Muphrid Jan 22 '13 at 4:00
@Muphrid yes thanks! – Elements Jan 24 '13 at 20:54
up vote 1 down vote accepted

Let's consider a more general case. Let $a$ be a vector in $\mathbb R^m$. You basically want,

$$a \cdot \nabla_\lambda [Y-\underline X(\lambda)]^2$$

where $\nabla_\lambda$ is a differential operator corresponding to derivatives with respect to $\lambda$. The chain rule tells that this is equal to

$$(a \cdot \nabla_\lambda [Y-\underline X(\lambda)] )\cdot \nabla_{Y-\underline X(\lambda)} [Y-\underline X(\lambda)]^2$$

To evaluate this, you just need a couple results about directional derivatives. First, $a \cdot \nabla_x \underline T(x) = \underline T(a)$ for any vectors $a, x$ and linear operator $\underline T$. Second, $a \cdot \nabla_x x^2 = 2a\cdot x$. With all this in mind, the expression reduces to

$$-\underline X(a) \cdot 2[Y-\underline X(\lambda)] = -2a \cdot \overline X[Y-\underline X(\lambda)]$$

To pick out a particular $\lambda_j$, simply pick $a$ such that it is full of zero entries save for a 1 in the right place.

All this follows from the chain rule and a couple basic identities for vector derivatives. Edit: Overall, it's easier (in my opinion) to consider matrix operations abstractly as linear maps instead of as matrices proper. This is basically what I've done here. Multiplication of matrices just becomes composition of functions, and that's something we know how to attack via the chain rule.

share|cite|improve this answer

See the entry on Matrix Calculus in Wikipedia, or search for "matrix calculus" on the internet. In your particular case, $$ \frac{\partial (Y-X\lambda)^T (Y-X\lambda)}{\partial\lambda^T}=-2(Y-X\lambda)^T X $$ and hence the partial derivative w.r.t. $\lambda_j$ is the $j$-th entry of the row vector $-2(Y-X\lambda)^T X$.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.