0
$\begingroup$

Recall for a function $f: \mathbb{R}^n \to \mathbb{R}$, the Gradient is \begin{equation} \nabla f(\mathbf{x}) = \begin{bmatrix} \frac{\partial f(\mathbf{x})}{\partial x_1} & \frac{\partial f(\mathbf{x})}{\partial x_2} & \cdots & \frac{\partial f(\mathbf{x})}{\partial x_n} \end{bmatrix} \end{equation} and the Hessian is \begin{equation} Hf(\mathbf{x}) = \begin{bmatrix} \frac{\partial^2 f(\mathbf{x})}{(\partial x_1)^2} & \frac{\partial^2 f(\mathbf{x})}{\partial x_1 \partial x_2} & \cdots & \frac{\partial^2 f(\mathbf{x})}{\partial x_1 \partial x_n} \\ \frac{\partial^2 f(\mathbf{x})}{\partial x_2\partial x_1} & \frac{\partial^2 f(\mathbf{x})}{(\partial x_2)^2} & \cdots & \frac{\partial^2 f(\mathbf{x})}{\partial x_2 \partial x_n} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial^2 f(\mathbf{x})}{\partial x_n\partial x_1} & \frac{\partial^2 f(\mathbf{x})}{\partial x_n\partial x_2} & \cdots & \frac{\partial^2 f(\mathbf{x})}{(\partial x_n)^2} \end{bmatrix} \end{equation} To me, this looks like an outer product matrix. So, this brings me to my question: given the gradient of $f$, if I wanted to compute the Hessian, is it true that \begin{align} Hf(\mathbf{x})&= \begin{bmatrix} \frac{\partial f(\mathbf{x})}{\partial x_1} \\ \frac{\partial f(\mathbf{x})}{\partial x_2} \\ \vdots \\ \frac{\partial f(\mathbf{x})}{\partial x_n} \end{bmatrix} \begin{bmatrix} \frac{\partial f(\mathbf{x})}{\partial x_1} & \frac{\partial f(\mathbf{x})}{\partial x_2} & \cdots & \frac{\partial f(\mathbf{x})}{\partial x_n} \end{bmatrix} \\ &=\nabla f(\mathbf{x})^{\top}\nabla f(\mathbf{x}) \\ &=sf(\mathbf{x})sf(\mathbf{x})^{\top} \end{align} where $sf(\mathbf{x})=\nabla f(\mathbf{x})^{\top}$ denotes the score? Thank you!

$\endgroup$
3
  • 1
    $\begingroup$ It would be more correct to write $H=\nabla^T\nabla$, where the differentiation operator themselves would be multiplied as opposed to their applications to $f$. $\endgroup$ Commented Nov 10, 2014 at 17:17
  • $\begingroup$ Aha! Okay, thanks. Is it also true that the Hessian is equal to the Jacobian of the score, so that $Hf(\mathbf{x})=\nabla sf(\mathbf{x})$? $\endgroup$ Commented Nov 12, 2014 at 6:54
  • $\begingroup$ The Hessian is the Jacobian of the Gradient, but the Jacobian is normally written as $J$. See en.wikipedia.org/wiki/Hessian_matrix . $\endgroup$ Commented Nov 12, 2014 at 18:05

2 Answers 2

3
$\begingroup$

I think you're just confused by notation.

$\frac{\partial^2 f}{\partial x_1 \partial x_2}(\mathbf{x}) \neq \frac{\partial f}{\partial x_1}(\mathbf{x}) \cdot \frac{\partial f}{\partial x_2}(\mathbf{x})$

$\endgroup$
1
  • $\begingroup$ Thanks for pointing this out! For clarification, is the Hessian equal to the Jacobian of the score, so that $Hf(\mathbf{x})=\nabla sf(\mathbf{x})$? $\endgroup$ Commented Nov 12, 2014 at 6:53
1
$\begingroup$

Note:

$\frac{\partial^2 f(\mathbf{x})}{\partial x_2 \partial x_1} \neq (\frac{\partial f(\mathbf{x})}{\partial x_2})*(\frac{\partial f(\mathbf{x})}{\partial x_1})$

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .