1
$\begingroup$

I am trying to find the Hessian of the following cost function for the logistic regression: $$ J(\theta) = \frac{1}{m}\sum_{i=1}^{m}\log(1+\exp(-y^{(i)}\theta^{T}x^{(i)}) $$ I intend to use this to implement Newton's method and update $\theta$, such that $$ \theta_{new} := \theta_{old} - H^{-1}\nabla_{\theta}J(\theta) $$ However, I am finding it rather difficult to obtain a convincing solution. Here's my effort at computing the gradient with respect to the vector $\theta$: \begin{align*} \nabla_{\theta}J(\theta) &= \frac{\partial}{\partial \theta_j}\left[\frac{1}{m}\sum_{i=1}^{m}\log(1+\exp(-y^{(i)}\theta^{T}x^{(i)})\right]\\ &= \frac{1}{m}\sum_{i=1}^{m}\frac{-y^{(i)}x^{(i)}_j \exp(-y^{(i)}\theta^T x^{(i)})}{1+\exp(-y^{(i)}\theta^T x^{(i)})} \end{align*} This result seems reasonable. If I go on and try to compute the second derivative, I get \begin{align*} \frac{\partial^2 J(\theta)}{\partial \theta_j \partial \theta_k} &= \frac{1}{m}\sum_{i=1}^m\frac{y^{(i)2}x^{(i)}_j x^{(i)}_k\cdot\left[\exp(-y^{(i)}\theta^Tx^{(i)}) + 2\exp(-2y^{(i)}\theta^Tx^{(i)})\right]}{\left[1 + \exp(-y^{(i)}\theta^Tx^{(i)}\right]^2} \end{align*}

where I obtained this result using the quotient formula.

Am I missing something obvious when it comes to simplifying this expression, or have I made an error in the differentiation?

$\endgroup$
2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.