0
$\begingroup$

Recall for a function $f: \mathbb{R}^n \to \mathbb{R}$, the Gradient is \begin{equation} \nabla f(\mathbf{x}) = \begin{bmatrix} \frac{\partial f(\mathbf{x})}{\partial x_1} & \frac{\partial f(\mathbf{x})}{\partial x_2} & \cdots & \frac{\partial f(\mathbf{x})}{\partial x_n} \end{bmatrix} \end{equation} and the Hessian is \begin{equation} Hf(\mathbf{x}) = \begin{bmatrix} \frac{\partial^2 f(\mathbf{x})}{(\partial x_1)^2} & \frac{\partial^2 f(\mathbf{x})}{\partial x_1 \partial x_2} & \cdots & \frac{\partial^2 f(\mathbf{x})}{\partial x_1 \partial x_n} \\ \frac{\partial^2 f(\mathbf{x})}{\partial x_2\partial x_1} & \frac{\partial^2 f(\mathbf{x})}{(\partial x_2)^2} & \cdots & \frac{\partial^2 f(\mathbf{x})}{\partial x_2 \partial x_n} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial^2 f(\mathbf{x})}{\partial x_n\partial x_1} & \frac{\partial^2 f(\mathbf{x})}{\partial x_n\partial x_2} & \cdots & \frac{\partial^2 f(\mathbf{x})}{(\partial x_n)^2} \end{bmatrix} \end{equation} To me, this looks like an outer product matrix. So, this brings me to my question: given the gradient of $f$, if I wanted to compute the Hessian, is it true that \begin{align} Hf(\mathbf{x})&= \begin{bmatrix} \frac{\partial f(\mathbf{x})}{\partial x_1} \\ \frac{\partial f(\mathbf{x})}{\partial x_2} \\ \vdots \\ \frac{\partial f(\mathbf{x})}{\partial x_n} \end{bmatrix} \begin{bmatrix} \frac{\partial f(\mathbf{x})}{\partial x_1} & \frac{\partial f(\mathbf{x})}{\partial x_2} & \cdots & \frac{\partial f(\mathbf{x})}{\partial x_n} \end{bmatrix} \\ &=\nabla f(\mathbf{x})^{\top}\nabla f(\mathbf{x}) \\ &=sf(\mathbf{x})sf(\mathbf{x})^{\top} \end{align} where $sf(\mathbf{x})=\nabla f(\mathbf{x})^{\top}$ denotes the score? Thank you!

$\endgroup$
  • $\begingroup$ It would be more correct to write $H=\nabla^T\nabla$, where the differentiation operator themselves would be multiplied as opposed to their applications to $f$. $\endgroup$ – Solomonoff's Secret Nov 10 '14 at 17:17
  • $\begingroup$ Aha! Okay, thanks. Is it also true that the Hessian is equal to the Jacobian of the score, so that $Hf(\mathbf{x})=\nabla sf(\mathbf{x})$? $\endgroup$ – Mathemanic Nov 12 '14 at 6:54
  • $\begingroup$ The Hessian is the Jacobian of the Gradient, but the Jacobian is normally written as $J$. See en.wikipedia.org/wiki/Hessian_matrix . $\endgroup$ – Solomonoff's Secret Nov 12 '14 at 18:05
1
$\begingroup$

I think you're just confused by notation.

$\frac{\partial^2 f}{\partial x_1 \partial x_2}(\mathbf{x}) \neq \frac{\partial f}{\partial x_1}(\mathbf{x}) \cdot \frac{\partial f}{\partial x_2}(\mathbf{x})$

$\endgroup$
  • $\begingroup$ Thanks for pointing this out! For clarification, is the Hessian equal to the Jacobian of the score, so that $Hf(\mathbf{x})=\nabla sf(\mathbf{x})$? $\endgroup$ – Mathemanic Nov 12 '14 at 6:53
1
$\begingroup$

Note:

$\frac{\partial^2 f(\mathbf{x})}{\partial x_2 \partial x_1} \neq (\frac{\partial f(\mathbf{x})}{\partial x_2})*(\frac{\partial f(\mathbf{x})}{\partial x_1})$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.