I'm trying to show that the Hessian of the negative of the log likelihood with two parameters is positive definite, but I'm not sure how to go about it once I compute the Hessian.

The function is: $-log L(\alpha) =-\sum\limits_{i=1}^N ((1-y_i)(-\alpha_0 - \alpha_1x_i) - log(1 + e^{-\alpha_0 - \alpha_1x_i}))$

and I believe that the Hessian is:

\begin{bmatrix} \frac{e^{-\alpha_{0}-\alpha{1}x_i}}{(e^{-\alpha_{0}-\alpha{1}x_i}+1)^2}&\frac{e^{-\alpha_{0}-\alpha{1}x_i}x_i}{(e^{-\alpha_{0}-\alpha{1}x_i}+1)^2}\\\frac{e^{-\alpha_{0}-\alpha{1}x_i}x_i}{(e^{-\alpha_{0}-\alpha{1}x_i}+1)^2}&\frac{e^{-\alpha_{0}-\alpha{1}x_i}x_i^2}{(1+e^{-\alpha_{0}-\alpha{1}x_i})^2}\end{bmatrix}

How can I show that this matrix is positive definite for all $\alpha$ in order to prove convexity?

  • $\begingroup$ factor out the (nonnegative) denominator in front and look at the trace and determinant $\endgroup$ – LinAlg Oct 17 '18 at 18:26
  • $\begingroup$ Hi, @LinAlg, I took your advice and got this: $det(H) = ac − b^2 = {e^{-\alpha_{0}-\alpha_{1}x_i}}{e^{-\alpha_{0}-\alpha_{1}x_i}x_i^2} - ({e^{-\alpha_{0}-\alpha_{1}x_i}x_i})^2 = e^{-2\alpha_{0}-2\alpha_{1}x_i}x^2 - e^{-2\alpha_{0}-2\alpha_{1}x_i}x^2 = 0$. Don't I need $det(H) > 0$ for it to be positive definite? $\endgroup$ – Miguel Barrera Oct 18 '18 at 1:05

The Hessian simplifies to: $$\frac{e^{-\alpha_{0}-\alpha{1}x_i}}{(1+e^{-\alpha_{0}-\alpha{1}x_i})^2}\begin{bmatrix} 1 & x_i\\x_i & x_i^2\end{bmatrix}.$$ The factor is positive and does not affect positive (semi)definiteness. The matrix has trace $1+x_i^2$ and determinant $0$. Therefore, the eigenvalues are $0$ and $1+x_i^2$. This matrix is therefore positive semidefinite.

  • $\begingroup$ Ahh, I see. I got the same thing, just by factoring out less. However, since I'm trying to prove convexity, don't I need the matrix to be positive definite and not just semidefinite? $\endgroup$ – Miguel Barrera Oct 18 '18 at 1:31
  • 1
    $\begingroup$ @MiguelBarrera No you do not. The function is convex. $\endgroup$ – LinAlg Oct 18 '18 at 2:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.