Given standard matrix inner product, \begin{equation} \begin{aligned} f(\textbf{X}) := & \;\;\;\; {\langle}{\textbf{X}, \textbf{A}\textbf{X}}{\rangle}\\ =& \; \text{tr} (\textbf{X}^{\text{T}} \textbf{A}\textbf{X}),\\ & \textbf{X} \in \mathcal{R}^{n \times r}, \text{ the variable,}\\ & \textbf{A} \in \mathcal{R}^{n \times n}, \text{ a constant matrix and isn't necessarily be symmetric}. \end{aligned} \end{equation}

I want to calculate the Hessian with respect to ${\textbf{X}}$ which is a matrix not a vector. I know how to compute the gradient which is,

\begin{equation} \begin{aligned} & \nabla_\textbf{X} f(\textbf{X})= \begin{cases} {\textbf{2AX}, \; \; \; \; \; \; \; \; \; \text{ if } \textbf{A} = \textbf{A}^{\text{T}}},\\ {\textbf{(A+A}^{\text{T}}) \textbf{X}, \text{ else. }} \end{cases}\\ & \in \mathcal{R}^{n \times r} \end{aligned} \end{equation}

but it is highly unclear how I do for a matrix variable. And in general, it is hard to find a material available and clear for this. Of course, the definition is in Wikipedia but for a matrix, I need a small example like $\textbf{X} \in \mathcal{R}^{3 \times 2}, \; \textbf{A} \in \mathcal{R}^{3 \times 3},$ then it will become clear.

In this case of the small example, the dimension of the Hessian Matrix will become $\textbf{X} \in \mathcal{R}^{6 \times 6}$ as far as believe.

And hopefully, there will exist $\textbf{the neat mathematical expression to denote the resulting hessian matrix}$ for this function as it does for the gradient.

$\textbf{With a clear example please}$, thanks in advance.

It will definitely help many people because this is fundamental but not well accessible.


Since you know how to calculate the gradient, let's start by taking the differential of that $$\eqalign{ S &= A+A^T \cr G &= \nabla f = SX \cr dG &= S\,dX \cr }$$ There are two ways to proceed: vectorize the equation or use tensors.

Vectorization flattens the $(dG,dX)$ matrices into vectors and the Hessian into a matrix. $$\eqalign{ dg &= (I\otimes S)\,dx \cr H = \frac{\partial g}{\partial x} &= I\otimes S \cr }$$ where $\otimes$ represents the Kronecker product and $\,\,dx={\rm vec}(dX)$.

But the true Hessian is a fourth-order tensor. $$\eqalign{ dG &= S{\mathcal E}:dX \cr {\mathcal H} = \frac{\partial G}{\partial X} &= S{\mathcal E} \cr }$$ where ${\mathcal E}$ is a tensor constant whose components can be written in terms of Kronecker deltas $$\eqalign{ {\mathcal E}_{ijkl} &= \delta_{ik}\delta_{jl} \cr }$$ The colon represents the double-contraction product $$B={\mathcal E}:X \implies B_{ij}=\sum_k\sum_l {\mathcal E}_{ijkl}\,X_{kl}$$ while juxtaposition represents the single-contraction product.

The components of the Hessian are equal to $$ {\mathcal H}_{ijkl} = \frac{\partial G_{ij}}{\partial X_{kl}} = \sum_nS_{in}{\mathcal E}_{njkl} = S_{ik}\delta_{jl} $$

  • $\begingroup$ Thanks a lot greg. The point is, I didn't have chance to study Tensor so far while it is becoming necessary slowly. Then the question that I want to ask at the moment is, [1] "Why the true Hessian is a fourth-order tensor?" and, I wanted to compute the Hessian of this function to check in which case, the resulting Hessian becomes Positive Definite. [2] "Can I work for this purpose on the linearized Hessian rather than 4th-order Tensor, like if the linearized Hessian is P.D., than there is some analogy to P.D. in the Tensor?". Thanks. $\endgroup$ – Robin Oct 25 '18 at 12:09
  • $\begingroup$ Ah, one more thing. What is the dimension of the Identity Matrix in the linearized Hessian (1st case)? Because in the Kronecker Product, the operands don't necessarily have the same dimensions. Thanks. $\endgroup$ – Robin Oct 25 '18 at 13:48
  • 1
    $\begingroup$ The identity matrix is $I\in{\mathbb R}^{r\times r}$ $\endgroup$ – greg Oct 26 '18 at 14:57
  • $\begingroup$ thanks, there seems to be a progress :) $\endgroup$ – Robin Oct 27 '18 at 14:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.