[Context: I'm working through problem set one of Andrew Ng's Machine Learning course, question 2, trying to derive the Hessian myself. My trouble, though, is with the simple linear algebra. I'll ignore the regularization parameters in my question to keep things simple.]

I can find $H_{jk}$ by taking partial derivatives of the original function with respect to $j$ and $k$. I've worked it through to the point where I know that $H_{jk} = \sum_{i=1}^m d_i X_{ij} X_{ik}$, and I know the answer is that $H = X^TDX$.

($d$ is a vector of reals, $H$, $X$ and $D$ are $m \times m$ matrices, $D$ diagonal with $D_{ii} = d_i$.)

It's easy for me to verify that $H = X^TDX$ gives $H_{jk} = \sum_{i=1}^m d_i X_{ij} X_{ik}$. But I don't "see" how to go easily in the opposite direction.

1) Is there a method, or a set of rules to memorize, or a way of talking through it that would make it easy, or do I just need to do enough basic linear algebra homework to start to recognize patterns?

2) Am I doing it wrong by breaking it down and deriving $H_{jk}$? Are there tricks for taking these kind of derivatives (which involve logs and exponentials) over the whole matrix at once?

  • $\begingroup$ There is only one pattern you need to recognize, and it's what a product of two matrices looks like in coordinates. And you shouldn't memorize this pattern; you should understand how it follows from the fact that matrices model linear transformations. $\endgroup$ Commented Feb 25, 2011 at 20:46
  • $\begingroup$ I recognize the product of two matrices in coordinates, but that didn't help me here. $\endgroup$
    – zellyn
    Commented Feb 25, 2011 at 21:56

1 Answer 1


I believe I have found the answer. The key insight is that multiplication by diagonal matrices multiplies each row $i$ by $D_{ii}$ when the diagonal matrix is on the left, and each column $i$ by $D_{ii}$ when the diagonal matrix is on the right.

Thus $\sum_{i=1}^m d_i X_{ij} X_{ik}$ can be rewritten as $\sum_{i=1}^m X_{ij} (DX)_{ik}$ = $\sum_{i=1}^m X_{ji}^T (DX)_{ik}$, which is a simple product of $X^T$ and $(DX)$.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .