[Context: I'm working through problem set one of Andrew Ng's Machine Learning course, question 2, trying to derive the Hessian myself. My trouble, though, is with the simple linear algebra. I'll ignore the regularization parameters in my question to keep things simple.]
I can find $H_{jk}$ by taking partial derivatives of the original function with respect to $j$ and $k$. I've worked it through to the point where I know that $H_{jk} = \sum_{i=1}^m d_i X_{ij} X_{ik}$, and I know the answer is that $H = X^TDX$.
($d$ is a vector of reals, $H$, $X$ and $D$ are $m \times m$ matrices, $D$ diagonal with $D_{ii} = d_i$.)
It's easy for me to verify that $H = X^TDX$ gives $H_{jk} = \sum_{i=1}^m d_i X_{ij} X_{ik}$. But I don't "see" how to go easily in the opposite direction.
1) Is there a method, or a set of rules to memorize, or a way of talking through it that would make it easy, or do I just need to do enough basic linear algebra homework to start to recognize patterns?
2) Am I doing it wrong by breaking it down and deriving $H_{jk}$? Are there tricks for taking these kind of derivatives (which involve logs and exponentials) over the whole matrix at once?
