1
$\begingroup$

Suppose $f: \mathbb{R^n} \to \mathbb{R}$, the gradient of $f(\mathbf{x})$ is $$\mathop{\nabla} f(\mathbf{x}) = \begin{bmatrix} \frac{\partial{f}}{\partial{x_1}} \\ \vdots \\ \frac{\partial{f}}{\partial{x_n}} \end{bmatrix}$$

The Jacobian matrix of $\mathop{\nabla} f(\mathbf{x})$ is $$\begin{align} \mathbf{D} (\mathop{\nabla} f(\mathbf{x})) &= \begin{bmatrix} \frac{{\partial^2}f}{{\partial}x_1^2} & \frac{{\partial^2}f}{{\partial}x_2{\partial}x_1} & \cdots & \frac{{\partial^2}f}{{\partial}x_n{\partial}x_1}\\ \frac{{\partial^2}f}{{\partial}x_1{\partial}x_2} & \frac{{\partial^2}f}{{\partial}x_2^2} & \cdots & \frac{{\partial^2}f}{{\partial}x_n{\partial}x_2}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{{\partial^2}f}{{\partial}x_1{\partial}x_n} & \frac{{\partial^2}f}{{\partial}x_2{\partial}x_n} & \cdots & \frac{{\partial^2}f}{{\partial}x_n^2}\\ \end{bmatrix} \\ &= \mathbf{H}^T \end{align}$$ where H is the Hessian matrix, which is consistent with the definition in Wikipedia.

The affine approximation of $\mathop{\nabla} f(\mathbf{x})$ around $x_n$ is $$\mathop{\nabla} f(\mathbf{x}) = \mathop{\nabla} f(\mathbf{x_n}) + \mathbf{D} (\mathop{\nabla} f(\mathbf{x_n})) (x - x_n) = \mathop{\nabla} f(\mathbf{x_n}) + \mathbf{H}^T (x - x_n)$$

Setting $\mathop{\nabla} f(\mathbf{x}) = 0$ gives the Newton-Raphson update as $$x_{n+1} := x_n - \mathbf{H}^{-T} \mathop{\nabla} f(\mathbf{x_n}) $$

However, in Wikipedia the Newton-Raphson update is given as $x_{n+1} := x_n - \mathbf{H}^{-1} \mathop{\nabla} f(\mathbf{x_n})$. The Hessian matrix is not symmetric if the entry of the matrix is not continuous. Did I do anything wrong with my calculation? If not, does this mean we can generally treat Hessian matrix as symmetric in practice for optimization?

$\endgroup$
1
$\begingroup$

From the first sentence of the Wikipedia article you link - "In optimization, Newton's method is applied to the derivative $f′$ of a twice-differentiable function $f$ to find the roots of the derivative." In other words, the Hessian is symmetric.

Newton's method can also be applied in a more general setting than optimization, to find roots of a differentiable function. In that case, there is no requirement that the Jacobian be symmetric.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.