4
$\begingroup$

Let $f:\mathbb{R}^n\rightarrow\mathbb{R}$ be a scalar function. Then, the gradient of $f(\mathbf{x})$ is defined as:

$$ \nabla_\mathbf{x} f(\mathbf{x}) = \begin{bmatrix} \frac{\partial f(\mathbf{x})}{\partial x_1} \\ \frac{\partial f(\mathbf{x})}{\partial x_2} \\ \vdots \\ \frac{\partial f(\mathbf{x})}{\partial x_n} \end{bmatrix} $$

However, if $f(\mathbf{x})$ is a scalar, then wouldn't the following also be valid?

$$ (\nabla_\mathbf{x}) (f(\mathbf{x})) = \begin{bmatrix} \frac{\partial}{\partial x_1} \\ \frac{\partial}{\partial x_2} \\ \vdots \\ \frac{\partial}{\partial x_n} \end{bmatrix} f(\mathbf{x}) = \begin{bmatrix} \frac{\partial f(\mathbf{x})}{\partial x_1} \\ \frac{\partial f(\mathbf{x})}{\partial x_2} \\ \vdots \\ \frac{\partial f(\mathbf{x})}{\partial x_n} \end{bmatrix} $$

In other words, the gradient $\nabla_\mathbf{x}$ is treated as a standalone vector, and then it is multiplied by the scalar $f(\mathbf{x})$. Interestingly, if this is indeed true, then I can build the Hessian matrix of $f(\mathbf{x})$ using the following outer product:

$$ (\nabla_x \nabla_x^T) (f(\mathbf{x})) $$

In other words, I left-multiply the row vector $\nabla_x^T$ by the column vector $\nabla_x$ to get an $n \times n$ matrix of second-order partial derivative operators. I then multiply this matrix by the scalar $f(\mathbf{x})$ to get the Hessian matrix. However, I am wondering if this is just a coincidence.

$\endgroup$
4
  • $\begingroup$ You can regard it as a matrix with one column with coefficients in a ring of operators on a certain function space (this is a vector of some vectorspace but not the one where your coefficients are the scalars). The product in that ring would be the composition of operators. That induces a matrix multiplication that makes your observation about the Hessian operator being the outer product true. You can also just see this as convenient notation which is often done. The inner product gives you the Laplace operator. $\endgroup$ Mar 25 at 22:53
  • $\begingroup$ The last part about multplying with the scalar $f(x)$ is not correct. Your coefficients are operators, so you could regard them as scalars, or you could regard real numbers as scalars, but not the functions that the operators act on. This is no problem though, you can just apply the resulting Hessian operator to $f$, just like it is done with the Laplace operator. $\endgroup$ Mar 25 at 22:55
  • $\begingroup$ Don't be confused by the fact that $f(x)$ is a real number, you never apply your operators (partial derivative, gradient, Hessian) on that number, you apply them on $f$ and then fill in $x$. $\endgroup$ Mar 25 at 22:59
  • $\begingroup$ @JensRenders Thanks a lot for the clarification! Would you like to turn this into an answer? If so, could you include references where I can read more about the theory behind this? Like the theory of operators? Also, is the gradient an operator just like the Laplace and Hessian operators? $\endgroup$
    – mhdadk
    Mar 25 at 23:02
2
$\begingroup$

This notation is often formally used, notably in the definition of the Laplace operator. We can make it explicit though.

For convenience, lets only work with functions that are $C^\infty$, such that all their partial derivatives exist and can again be partially differentiated and so on (think about the issues that happen when we don't do this).

A partial derivative is just a function (operator) that sends functions to functions: $$\frac{d}{dx_i}: C^\infty(\mathbb{R}^n, \mathbb{R}) \to C^\infty(\mathbb{R}^n, \mathbb{R})$$ We can regard the gradient as a $1 \times n$ matrix with such operators as coefficients, so the gradient is then an element of the set $C^\infty(\mathbb{R}^n, \mathbb{R})^{1\times n}$. This gradient, and any other matrix of operators, is then itself an operator that returns a (real) matrix, by applying the operators pointwise to a function.

To be able to multiply matrices with coefficients in $C^\infty(\mathbb{R}^n, \mathbb{R})$, we need to be able to multiply the coefficients with each other. The multiplication of operators can be defined as composition. In this setting we indeed have $$ \nabla^T \nabla = L$$ $$ \nabla \nabla^T = H$$ where $L$ is the Laplace operator and $H$ is the Hessian operator.

$\endgroup$
-1
$\begingroup$

$\nabla \nabla^T$ is not a common notation.

Although sometimes the notation $(\nabla \cdot \nabla) f = \nabla^2 f$ is used to denote $\Delta f$, where $\Delta$ is the Laplace operator, it's important to note that $$(\nabla \cdot \nabla) f = \Delta f = \operatorname{tr} H_f$$ results in the trace of the Hessian $H_f$ of $f$. Some consider this notation strongly misleading: The square of first order derivatives, as $(\nabla \cdot \nabla) f$ might suggest, does not equate to the second order derivative. After all, with $f = x^2$, the second order derivative is $f'' = 2$ and not $\left(f'\right)^2 = (2x)^2 = 4x^2$.

Your observation however has a common application in optimization. Second-order optimization methods use the Hessian to improve an initial guess $x_0$ of a minimum of $f$. Instead of calculating the exact Hessian at $x_0$, the Gauss–Newton algorithm approximates the Hessian using $$H_f(x_0) \approx J^T(x_0)\,J(x_0),$$ where $J$ is the Jacobian. This question discusses this approximation.

$\endgroup$
2
  • $\begingroup$ $(\nabla \cdot \nabla) f$ does not imply the square of first order derivatives, at least not in the same way that $f'^2$ does. If we write $f'=\text Df$ using operator notation, then $f'' = D^2f$ is perfectly valid. $\nabla f\cdot \nabla f$ is the proper analogy to $f'^2$, and similarly is not a second derivative. $\endgroup$
    – Alex Jones
    Mar 26 at 1:02
  • $\begingroup$ @AlexanderJ93 You must have misread my answer. I did not claim that $\nabla^2$ "implies" the square of first order derivatives. I simply pointed out that it can easily be misread in this sense. But I have updated my answer to avoid further confusion of imply and suggest. $\endgroup$
    – yavagi
    Mar 26 at 1:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.