Can the gradient operator $\nabla_\mathbf{x}$ be treated as a standalone vector?

Question

Let $f:\mathbb{R}^n\rightarrow\mathbb{R}$ be a scalar function. Then, the gradient of $f(\mathbf{x})$ is defined as:

$$ \nabla_\mathbf{x} f(\mathbf{x}) = \begin{bmatrix} \frac{\partial f(\mathbf{x})}{\partial x_1} \\ \frac{\partial f(\mathbf{x})}{\partial x_2} \\ \vdots \\ \frac{\partial f(\mathbf{x})}{\partial x_n} \end{bmatrix} $$

However, if $f(\mathbf{x})$ is a scalar, then wouldn't the following also be valid?

$$ (\nabla_\mathbf{x}) (f(\mathbf{x})) = \begin{bmatrix} \frac{\partial}{\partial x_1} \\ \frac{\partial}{\partial x_2} \\ \vdots \\ \frac{\partial}{\partial x_n} \end{bmatrix} f(\mathbf{x}) = \begin{bmatrix} \frac{\partial f(\mathbf{x})}{\partial x_1} \\ \frac{\partial f(\mathbf{x})}{\partial x_2} \\ \vdots \\ \frac{\partial f(\mathbf{x})}{\partial x_n} \end{bmatrix} $$

In other words, the gradient $\nabla_\mathbf{x}$ is treated as a standalone vector, and then it is multiplied by the scalar $f(\mathbf{x})$. Interestingly, if this is indeed true, then I can build the Hessian matrix of $f(\mathbf{x})$ using the following outer product:

$$ (\nabla_x \nabla_x^T) (f(\mathbf{x})) $$

In other words, I left-multiply the row vector $\nabla_x^T$ by the column vector $\nabla_x$ to get an $n \times n$ matrix of second-order partial derivative operators. I then multiply this matrix by the scalar $f(\mathbf{x})$ to get the Hessian matrix. However, I am wondering if this is just a coincidence.

Answer 1

This notation is often formally used, notably in the definition of the Laplace operator. We can make it explicit though.

For convenience, lets only work with functions that are $C^\infty$, such that all their partial derivatives exist and can again be partially differentiated and so on (think about the issues that happen when we don't do this).

A partial derivative is just a function (operator) that sends functions to functions: $$\frac{d}{dx_i}: C^\infty(\mathbb{R}^n, \mathbb{R}) \to C^\infty(\mathbb{R}^n, \mathbb{R})$$ We can regard the gradient as a $1 \times n$ matrix with such operators as coefficients, so the gradient is then an element of the set $C^\infty(\mathbb{R}^n, \mathbb{R})^{1\times n}$. This gradient, and any other matrix of operators, is then itself an operator that returns a (real) matrix, by applying the operators pointwise to a function.

To be able to multiply matrices with coefficients in $C^\infty(\mathbb{R}^n, \mathbb{R})$, we need to be able to multiply the coefficients with each other. The multiplication of operators can be defined as composition. In this setting we indeed have $$ \nabla^T \nabla = L$$ $$ \nabla \nabla^T = H$$ where $L$ is the Laplace operator and $H$ is the Hessian operator.

Answer 2

$\nabla \nabla^T$ is not a common notation.

Although sometimes the notation $(\nabla \cdot \nabla) f = \nabla^2 f$ is used to denote $\Delta f$, where $\Delta$ is the Laplace operator, it's important to note that $$(\nabla \cdot \nabla) f = \Delta f = \operatorname{tr} H_f$$ results in the trace of the Hessian $H_f$ of $f$. Some consider this notation strongly misleading: The square of first order derivatives, as $(\nabla \cdot \nabla) f$ might suggest, does not equate to the second order derivative. After all, with $f = x^2$, the second order derivative is $f'' = 2$ and not $\left(f'\right)^2 = (2x)^2 = 4x^2$.

Your observation however has a common application in optimization. Second-order optimization methods use the Hessian to improve an initial guess $x_0$ of a minimum of $f$. Instead of calculating the exact Hessian at $x_0$, the Gauss–Newton algorithm approximates the Hessian using $$H_f(x_0) \approx J^T(x_0)\,J(x_0),$$ where $J$ is the Jacobian. This question discusses this approximation.