For the second derivative I got $$\frac{\partial^2}{\partial x_k x_j}\log \left(\sum_{i=1}^{n} e^{x_i}\right)=-\frac{e^{x_k}e^{x_j}}{\left(\sum_{i=1}^{n} e^{x_i}\right)^2},$$ where $j \neq k$, and $1 \le j,k \le n$.

This Hessian is negative, which can't allow $\log(\sum_{i=1}^{n}{e^{x_i}})$ to be convex, but I am asked to show that $\log(\sum_{i=1}^{n}{e^{x_i}})$ is convex. Obviously something is wrong here, so unless the second derivative I showed is actually positive, then I must have just gotten the wrong Hessian.

Someone helped me calculate that:

$$\frac{\partial^2}{\partial x_k^2}\log \left(\sum_{i=1}^{n} e^{x_i}\right)=\frac{e^{x_k}\left(\sum_{i=1}^{n} e^{x_i}-e^{x_k}\right)}{\left(\sum_{i=1}^{n} e^{x_i}\right)^2},$$

  • $\begingroup$ where did the negative sign come from? $\endgroup$ – Jacob Claassen Sep 4 '17 at 20:03
  • 1
    $\begingroup$ nvm, I got it, but the terms on top should be added, not multiplied, yes? $\endgroup$ – Jacob Claassen Sep 4 '17 at 20:04
  • $\begingroup$ Even so, there is still a negative sign, so in the grand scheme of things, this shouldn't be convex. $\endgroup$ – dirtysocks45 Sep 4 '17 at 20:06
  • $\begingroup$ I see. So I should get rid of the negative sign? $\endgroup$ – dirtysocks45 Sep 4 '17 at 20:06
  • $\begingroup$ I did not use the eigenvalues of the Hessian. $\endgroup$ – dirtysocks45 Sep 4 '17 at 20:08

Simply a story of variance...

Using the short-hands $$t_i=s\cdot e^{x_i}\qquad s=\left(\sum_je^{x_j}\right)^{-1}$$ the diagonal and off-diagonal entries of the Hessian are $$t_i(1-t_i)\qquad\text{and}\qquad -t_it_j$$ respectively, hence it suffices to show that $Q(u)\geqslant0$ for every $u=(u_i)$, where $$Q(u)=\sum_i t_i(1-t_i)u_i^2- \sum_{i\ne j}t_it_ju_iu_j$$ But, by construction, $$\sum_it_i=1$$ hence $$Q(u)=\sum_it_iu_i^2\cdot\sum_it_i-\left(\sum_it_iu_i\right)^2$$ hence Cauchy-Schwarz inequality shows that indeed $Q(u)\geqslant0$.

  • $\begingroup$ If only I had the background to understand this. $\endgroup$ – dirtysocks45 Sep 4 '17 at 20:15
  • $\begingroup$ @dirtysocks45 Which part is unclear? $\endgroup$ – Did Sep 4 '17 at 20:16
  • $\begingroup$ I don't understand the point of $s$. $\endgroup$ – dirtysocks45 Sep 4 '17 at 20:19
  • $\begingroup$ To get correct formulas for the entries of the Hessian. Seems like a sufficient point to me... $\endgroup$ – Did Sep 4 '17 at 20:29

There is nothing wrong. What you have only shown is not that the Hessian matrix $H$ is negative definite, but merely that the off-diagonal entries of $H$ are negative. The matrix $\pmatrix{1&-1\\ -1&1}$, for instance, has negative off-diagonal entries, but the matrix itself is positive semidefinite.

In your case, it only takes a little more work to show that the $i$-th diagonal entry of the Hessian matrix $H$ is given by $$ \frac{e^{x_i}}{\sum_{i=1}^ne^{x_i}}-\left(\frac{e^{x_i}}{\sum_{i=1}^ne^{x_i}}\right)^2. $$ Therefore $H$ is a (weakly) diagonally dominant matrix with a nonnegative diagonal, meaning that its eigenvalues have nonnegative real parts (Gershgorin disc theorem). As $H$ is also real symmetric, it has real eigenvalues. Hence all its eigenvalues of $H$ are nonnegative, i.e. $H$ is positive semidefinite.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.