# Confusion related to calculation of Hessian

I want to know how the Hessian of the function

$f(x) = \log\sum_{k=1}^n\exp(x_k)$

comes out to be

$\frac{1}{\textbf{1}^Tz}\mathrm{diag}(z)-\frac{1}{(\textbf{1}^Tz)^2}zz^T$

given $z_k=\exp(x_k)$

• It's a straightforward calculation. Why don't you try it for the case $n=2$? That will show you why the result is a difference of two terms and where all the pieces are coming from. – whuber Jan 9 '13 at 23:56

## 1 Answer

We have $$f(x) = \log(\sum_k \exp(x_k))$$ Our first partial derivative is $$\partial x_i f (x) = \frac{\exp(x_i)}{\sum_k\exp(x_k)}$$ But we need second derivatives, therefore \begin{align} \partial x_i \partial x_i f (x)& = \frac{\exp(x_i)\cdot(\sum_{k,k\not =i} \exp(x_i))}{(\sum_k\exp(x_k))^2} \\ &= \frac{\exp(x_i)}{\sum_k\exp(x_k)} -\frac{\exp(2x_i)}{(\sum_k\exp(x_k))^2} \end{align} and \begin{align} \partial x_j \partial x_i f (x) = -\frac{\exp(x_i)\cdot\exp(x_j)}{(\sum_k\exp(x_k))^2} \end{align} Therefore we can rewrite our Hessian as \begin{align} H_f = \frac{1}{\sum_k\exp(x_k)} \begin{pmatrix} \exp(x_1) & 0 & 0 & ...\\ 0 & \exp(x_2) & 0 & ...\\ ... & ... & ... & ...\\ 0 & ... & 0 &\exp(x_n)\\ \end{pmatrix} - \frac{1}{(\sum_k\exp(x_k))^2} \begin{pmatrix} \exp(2x_1) & \exp(x_1x_2) & \exp(x_1x_3) & ...\\ \exp(x_2x_1) & \exp(2x_2) & \exp(x_2x_3) & ...\\ ... & ... & ... & ...\\ ... & ... & ... &\exp(2x_n)\\ \end{pmatrix} \end{align} which is exactly what you should show if we define $z := (\exp(x_1),...,\exp(x_n))^T$. Your task is now to proof that I calculated the corred deriatives and that you can split the second derivative for $\partial x_i\partial x_i f$ in two terms as i claimed.