# Connection of Gradient, Jacobian and Hessian matrix in Newton method

Suppose $f: \mathbb{R^n} \to \mathbb{R}$, the gradient of $f(\mathbf{x})$ is $$\mathop{\nabla} f(\mathbf{x}) = \begin{bmatrix} \frac{\partial{f}}{\partial{x_1}} \\ \vdots \\ \frac{\partial{f}}{\partial{x_n}} \end{bmatrix}$$

The Jacobian matrix of $\mathop{\nabla} f(\mathbf{x})$ is \begin{align} \mathbf{D} (\mathop{\nabla} f(\mathbf{x})) &= \begin{bmatrix} \frac{{\partial^2}f}{{\partial}x_1^2} & \frac{{\partial^2}f}{{\partial}x_2{\partial}x_1} & \cdots & \frac{{\partial^2}f}{{\partial}x_n{\partial}x_1}\\ \frac{{\partial^2}f}{{\partial}x_1{\partial}x_2} & \frac{{\partial^2}f}{{\partial}x_2^2} & \cdots & \frac{{\partial^2}f}{{\partial}x_n{\partial}x_2}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{{\partial^2}f}{{\partial}x_1{\partial}x_n} & \frac{{\partial^2}f}{{\partial}x_2{\partial}x_n} & \cdots & \frac{{\partial^2}f}{{\partial}x_n^2}\\ \end{bmatrix} \\ &= \mathbf{H}^T \end{align} where H is the Hessian matrix, which is consistent with the definition in Wikipedia.

The affine approximation of $\mathop{\nabla} f(\mathbf{x})$ around $x_n$ is $$\mathop{\nabla} f(\mathbf{x}) = \mathop{\nabla} f(\mathbf{x_n}) + \mathbf{D} (\mathop{\nabla} f(\mathbf{x_n})) (x - x_n) = \mathop{\nabla} f(\mathbf{x_n}) + \mathbf{H}^T (x - x_n)$$

Setting $\mathop{\nabla} f(\mathbf{x}) = 0$ gives the Newton-Raphson update as $$x_{n+1} := x_n - \mathbf{H}^{-T} \mathop{\nabla} f(\mathbf{x_n})$$

However, in Wikipedia the Newton-Raphson update is given as $x_{n+1} := x_n - \mathbf{H}^{-1} \mathop{\nabla} f(\mathbf{x_n})$. The Hessian matrix is not symmetric if the entry of the matrix is not continuous. Did I do anything wrong with my calculation? If not, does this mean we can generally treat Hessian matrix as symmetric in practice for optimization?

## 1 Answer

From the first sentence of the Wikipedia article you link - "In optimization, Newton's method is applied to the derivative $f′$ of a twice-differentiable function $f$ to find the roots of the derivative." In other words, the Hessian is symmetric.

Newton's method can also be applied in a more general setting than optimization, to find roots of a differentiable function. In that case, there is no requirement that the Jacobian be symmetric.