I am trying to find the Hessian of the following cost function for the logistic regression: $$ J(\theta) = \frac{1}{m}\sum_{i=1}^{m}\log(1+\exp(-y^{(i)}\theta^{T}x^{(i)}) $$ I intend to use this to implement Newton's method and update $\theta$, such that $$ \theta_{new} := \theta_{old} - H^{-1}\nabla_{\theta}J(\theta) $$ However, I am finding it rather difficult to obtain a convincing solution. Here's my effort at computing the gradient with respect to the vector $\theta$: \begin{align*} \nabla_{\theta}J(\theta) &= \frac{\partial}{\partial \theta_j}\left[\frac{1}{m}\sum_{i=1}^{m}\log(1+\exp(-y^{(i)}\theta^{T}x^{(i)})\right]\\ &= \frac{1}{m}\sum_{i=1}^{m}\frac{-y^{(i)}x^{(i)}_j \exp(-y^{(i)}\theta^T x^{(i)})}{1+\exp(-y^{(i)}\theta^T x^{(i)})} \end{align*} This result seems reasonable. If I go on and try to compute the second derivative, I get \begin{align*} \frac{\partial^2 J(\theta)}{\partial \theta_j \partial \theta_k} &= \frac{1}{m}\sum_{i=1}^m\frac{y^{(i)2}x^{(i)}_j x^{(i)}_k\cdot\left[\exp(-y^{(i)}\theta^Tx^{(i)}) + 2\exp(-2y^{(i)}\theta^Tx^{(i)})\right]}{\left[1 + \exp(-y^{(i)}\theta^Tx^{(i)}\right]^2} \end{align*}

where I obtained this result using the quotient formula.

Am I missing something obvious when it comes to simplifying this expression, or have I made an error in the differentiation?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.