4
$\begingroup$

Do I have the correct solution for the second derivative of the cost function of a logistic function?

Cost Function $$J(\theta)=-\frac{1}{m}\sum_{i=1}^{m}y^{i}\log(h_\theta(x^{i}))+(1-y^{i})\log(1-h_\theta(x^{i}))$$

where $h_{\theta}(x)$ is defined as follows

$$h_{\theta}(x)=g(\theta^{T}x)$$ $$g(z)=\frac{1}{1+e^{-z}}$$

First Derivative $$ \frac{\partial}{\partial\theta_{j}}J(\theta) =\sum_{i=1}^{m}(h_\theta(x^{i})-y^i)x_j^i$$

Second Derivative $$ \begin{align*} \frac{\partial}{\partial^2\theta_{j}}J'(\theta) &= \frac{\partial}{\partial\theta}\sum_{i=1}^{m}(h_\theta(x^{i})x_j^i -y^ix^i_j) \\ &= \frac{\partial}{\partial\theta}\sum_{i=1}^{m}(h_\theta(x^{i})x_j^i) \\ &= \frac{\partial}{\partial\theta}\sum_{i=1}^{m}\frac{x^{i}}{1+e^{-z}} \\ &= x^2 h_\theta(x) ^2 \end{align*} $$

$\endgroup$

1 Answer 1

4
$\begingroup$

For convenience, define some variables and their differentials $$\eqalign{ z &= X^T\theta, &\,\,\,\, dz = X^Td\theta \cr p &= \exp(z), &\,\,\,\, dp = p\odot dz \cr h &= \frac{p}{1+p}, &\,\,\,\, dh = (h-h\odot h)\odot dz \,= (H-H^2)\,dz \cr }$$ where
$\,\,\,\,H={\rm Diag}(h)$
$\,\,\,\,\odot$ represents the Hadamard elementwise product
$\,\,\,\,\exp$ is applied elementwise
$\,\,\,\frac{p}{1+p}$ represents elementwise division


The cost function can be written in terms of these variables and the Frobenius inner product (represented by a colon) $$\eqalign{ J &= -\frac{1}{m}\Big[y:\log(h) + (1-y):\log(1-h)\Big] \cr }$$

The differential of the cost is $$\eqalign{ dJ &= -\frac{1}{m}\Big[y:d\log(h) + (1-y):d\log(1-h)\Big] \cr &= -\frac{1}{m}\Big[y:H^{-1}dh - (1-y):(I-H)^{-1}dh\Big] \cr &= -\frac{1}{m}\Big[H^{-1}y - (I-H)^{-1}(1-y)\Big]:dh \cr &= -\frac{1}{m}\Big[H^{-1}y - (I-H)^{-1}(1-y)\Big]:H(I-H)dz \cr &= -\frac{1}{m}\Big[(I-H)y - H(1-y)\Big]:dz \cr &= -\frac{1}{m}(y-h):X^Td\theta \cr &= \frac{1}{m}X(h-y):d\theta \cr }$$ The gradient $$\eqalign{ G =\frac{\partial J}{\partial\theta} &= \frac{1}{m}X(h-y) \cr }$$ (NB: Your gradient is missing the $\frac{1}{m}$ factor)

The differential of the gradient $$\eqalign{ dG &= \frac{1}{m}X\,dh \cr &= \frac{1}{m}X(H-H^2)\,dz \cr &= \frac{1}{m}X(H-H^2)X^T\,d\theta \cr\cr }$$ And finally, the gradient of the gradient (aka the Hessian) $$\eqalign{ \frac{\partial^2J}{\partial\theta\,\partial\theta^T} &= \frac{\partial G}{\partial\theta} = \frac{1}{m}X(H-H^2)X^T \cr }$$

$\endgroup$
1
  • $\begingroup$ by Diag(h) you mean Diagonal Matrix of h ? $\endgroup$
    – guru_007
    Aug 10, 2019 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.