I was hoping that someone could verify the steps of computing a Hessian matrix.

I have the following function, $F:\mathbb{R}^n\to\mathbb{R}$,

$$F({\bf x}) = \sum_{i=1}^mf(g(A_i^T{\bf x}))$$

where each $A_i\in\mathbb{R}^{n\times n_i}$ translates ${\bf x}$ into some subvector of ${\bf x}$, for example, $A_1^T{\bf x} = (x_1,x_5,x_6,x_{10})$ (Notice that $A_i$ just consists of zeros and ones). Other functions are defined as $f:\mathbb{R}\to\mathbb{R}$ and $g:\mathbb{R}^{n_i}\to\mathbb{R}$.

I want to determine the Hessian, denoted $\nabla^2_{{\bf x}} F({\bf x})$.


I have computed the gradient to be \begin{align*} \nabla_{\bf x}F({\bf x}) &= \sum_{i=1}^m \frac{\partial}{\partial {\bf x}}f(g(A_i^T{\bf x})) \end{align*} which component-wise is \begin{align*} [\nabla_{\bf x}F({\bf x})]_k&= \sum_{i=1}^mf'(g(A_i^T{\bf x}))\sum_{j=1}^{n_i}\frac{\partial g({\bf v})}{\partial v_j}\frac{\partial v_j}{\partial x_k}\\ \end{align*} where ${\bf v} = A_i^T{\bf x}$.

Is the work so far correct? It seems to be dimensionally consistent. The term, $\frac{\partial v_j}{\partial x_k} = \frac{\partial [A_i^T{\bf x}]_j}{\partial x_k}$, can be simplified to read \begin{align*} \frac{\partial v_j}{\partial x_k} = \frac{\partial [A_i^T{\bf x}]_j}{\partial x_k} = [A_i{\bf e}_j]_k \end{align*} where ${\bf e}_j\in\mathbb{R}^{n}$ is a zero column vector with a $1$ in the $j^{th}$ position.


To start, the Hessian is given by the expression

\begin{align*} \nabla^2_{\bf x}F({\bf x}) &= \sum_{i=1}^m \frac{\partial^2}{\partial {\bf x}^2}f(g(A_i^T{\bf x})) \end{align*}

which component-wise, differentiating the gradient, becomes

\begin{align*} [\nabla^2_{\bf x}F({\bf x})]_{kl}&= \frac{\partial}{\partial x_l}\sum_{i=1}^mf'(g(A_i^T{\bf x}))\sum_{j=1}^{n_i}\frac{\partial g({\bf v})}{\partial v_j}[A_i{\bf e}_j]_k\\ &= \sum_{i=1}^m\frac{\partial f'(g(A_i^T{\bf x}))}{\partial x_l}\sum_{j=1}^{n_i}\frac{\partial g({\bf v})}{\partial v_j}[A_i{\bf e}_j]_k\\ &\qquad\qquad+ \sum_{i=1}^mf'(g(A_i^T{\bf x}))\sum_{j=1}^{n_i}\frac{\partial}{\partial x_l}\left[\frac{\partial g({\bf v})}{\partial v_j}[A_i{\bf e}_j]_k\right]. \end{align*}

Notice that the term $\frac{\partial g({\bf v})}{\partial v_j}$ is a function of ${\bf x}$. So, applying the product rule/chain rule, we obtain \begin{align*} [\nabla^2_{\bf x}F({\bf x})]_{kl}&= \sum_{i=1}^mf''(g(A_i^T{\bf x}))\left(\sum_{j=1}^{n_i}\frac{\partial g({\bf v})}{\partial v_j}[A_i{\bf e}_j]_k\right)\left(\sum_{j=1}^{n_i}\frac{\partial g({\bf v})}{\partial v_j}[A_i{\bf e}_j]_l\right)\\ &\qquad+\sum_{i=1}^mf'(g(A_i^T{\bf x}))\sum_{j=1}^{n_i}\frac{\partial^2 g({\bf v})}{\partial x_l\partial v_j}[A_i{\bf e}_j]_k \end{align*} where the term $\frac{\partial^2 g({\bf v})}{\partial x_l\partial v_j}$ is only non-zero if $[A_i^T{\bf x}]_j = x_l$.

Is the above work correct? Is there any way to verify its correctness other than rechecking each step?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.