Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

[Context: I'm working through problem set one of Andrew Ng's Machine Learning course, question 2, trying to derive the Hessian myself. My trouble, though, is with the simple linear algebra. I'll ignore the regularization parameters in my question to keep things simple.]

I can find $H_{jk}$ by taking partial derivatives of the original function with respect to $j$ and $k$. I've worked it through to the point where I know that $H_{jk} = \sum_{i=1}^m d_i X_{ij} X_{ik}$, and I know the answer is that $H = X^TDX$.

($d$ is a vector of reals, $H$, $X$ and $D$ are $m \times m$ matrices, $D$ diagonal with $D_{ii} = d_i$.)

It's easy for me to verify that $H = X^TDX$ gives $H_{jk} = \sum_{i=1}^m d_i X_{ij} X_{ik}$. But I don't "see" how to go easily in the opposite direction.

1) Is there a method, or a set of rules to memorize, or a way of talking through it that would make it easy, or do I just need to do enough basic linear algebra homework to start to recognize patterns?

2) Am I doing it wrong by breaking it down and deriving $H_{jk}$? Are there tricks for taking these kind of derivatives (which involve logs and exponentials) over the whole matrix at once?

share|cite|improve this question
There is only one pattern you need to recognize, and it's what a product of two matrices looks like in coordinates. And you shouldn't memorize this pattern; you should understand how it follows from the fact that matrices model linear transformations. – Qiaochu Yuan Feb 25 '11 at 20:46
I recognize the product of two matrices in coordinates, but that didn't help me here. – zellyn Feb 25 '11 at 21:56

I believe I have found the answer. The key insight is that multiplication by diagonal matrices multiplies each row $i$ by $D_{ii}$ when the diagonal matrix is on the left, and each column $i$ by $D_{ii}$ when the diagonal matrix is on the right.

Thus $\sum_{i=1}^m d_i X_{ij} X_{ik}$ can be rewritten as $\sum_{i=1}^m X_{ij} (DX)_{ik}$ = $\sum_{i=1}^m X_{ji}^T (DX)_{ik}$, which is a simple product of $X^T$ and $(DX)$.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.