1
$\begingroup$

I was hoping that someone could verify the steps of computing a Hessian matrix.

I have the following function, $F:\mathbb{R}^n\to\mathbb{R}$,

$$F({\bf x}) = \sum_{i=1}^mf(g(A_i^T{\bf x}))$$

where each $A_i\in\mathbb{R}^{n\times n_i}$ translates ${\bf x}$ into some subvector of ${\bf x}$, for example, $A_1^T{\bf x} = (x_1,x_5,x_6,x_{10})$ (Notice that $A_i$ just consists of zeros and ones). Other functions are defined as $f:\mathbb{R}\to\mathbb{R}$ and $g:\mathbb{R}^{n_i}\to\mathbb{R}$.

I want to determine the Hessian, denoted $\nabla^2_{{\bf x}} F({\bf x})$.

Gradient

I have computed the gradient to be \begin{align*} \nabla_{\bf x}F({\bf x}) &= \sum_{i=1}^m \frac{\partial}{\partial {\bf x}}f(g(A_i^T{\bf x})) \end{align*} which component-wise is \begin{align*} [\nabla_{\bf x}F({\bf x})]_k&= \sum_{i=1}^mf'(g(A_i^T{\bf x}))\sum_{j=1}^{n_i}\frac{\partial g({\bf v})}{\partial v_j}\frac{\partial v_j}{\partial x_k}\\ \end{align*} where ${\bf v} = A_i^T{\bf x}$.

Is the work so far correct? It seems to be dimensionally consistent. The term, $\frac{\partial v_j}{\partial x_k} = \frac{\partial [A_i^T{\bf x}]_j}{\partial x_k}$, can be simplified to read \begin{align*} \frac{\partial v_j}{\partial x_k} = \frac{\partial [A_i^T{\bf x}]_j}{\partial x_k} = [A_i{\bf e}_j]_k \end{align*} where ${\bf e}_j\in\mathbb{R}^{n}$ is a zero column vector with a $1$ in the $j^{th}$ position.

Hessian

To start, the Hessian is given by the expression

\begin{align*} \nabla^2_{\bf x}F({\bf x}) &= \sum_{i=1}^m \frac{\partial^2}{\partial {\bf x}^2}f(g(A_i^T{\bf x})) \end{align*}

which component-wise, differentiating the gradient, becomes

\begin{align*} [\nabla^2_{\bf x}F({\bf x})]_{kl}&= \frac{\partial}{\partial x_l}\sum_{i=1}^mf'(g(A_i^T{\bf x}))\sum_{j=1}^{n_i}\frac{\partial g({\bf v})}{\partial v_j}[A_i{\bf e}_j]_k\\ &= \sum_{i=1}^m\frac{\partial f'(g(A_i^T{\bf x}))}{\partial x_l}\sum_{j=1}^{n_i}\frac{\partial g({\bf v})}{\partial v_j}[A_i{\bf e}_j]_k\\ &\qquad\qquad+ \sum_{i=1}^mf'(g(A_i^T{\bf x}))\sum_{j=1}^{n_i}\frac{\partial}{\partial x_l}\left[\frac{\partial g({\bf v})}{\partial v_j}[A_i{\bf e}_j]_k\right]. \end{align*}

Notice that the term $\frac{\partial g({\bf v})}{\partial v_j}$ is a function of ${\bf x}$. So, applying the product rule/chain rule, we obtain \begin{align*} [\nabla^2_{\bf x}F({\bf x})]_{kl}&= \sum_{i=1}^mf''(g(A_i^T{\bf x}))\left(\sum_{j=1}^{n_i}\frac{\partial g({\bf v})}{\partial v_j}[A_i{\bf e}_j]_k\right)\left(\sum_{j=1}^{n_i}\frac{\partial g({\bf v})}{\partial v_j}[A_i{\bf e}_j]_l\right)\\ &\qquad+\sum_{i=1}^mf'(g(A_i^T{\bf x}))\sum_{j=1}^{n_i}\frac{\partial^2 g({\bf v})}{\partial x_l\partial v_j}[A_i{\bf e}_j]_k \end{align*} where the term $\frac{\partial^2 g({\bf v})}{\partial x_l\partial v_j}$ is only non-zero if $[A_i^T{\bf x}]_j = x_l$.

Is the above work correct? Is there any way to verify its correctness other than rechecking each step?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.