# Second derivative of the cost function of logistic function

Do I have the correct solution for the second derivative of the cost function of a logistic function?

Cost Function $$J(\theta)=-\frac{1}{m}\sum_{i=1}^{m}y^{i}\log(h_\theta(x^{i}))+(1-y^{i})\log(1-h_\theta(x^{i}))$$

where $h_{\theta}(x)$ is defined as follows

$$h_{\theta}(x)=g(\theta^{T}x)$$ $$g(z)=\frac{1}{1+e^{-z}}$$

First Derivative $$\frac{\partial}{\partial\theta_{j}}J(\theta) =\sum_{i=1}^{m}(h_\theta(x^{i})-y^i)x_j^i$$

Second Derivative \begin{align*} \frac{\partial}{\partial^2\theta_{j}}J'(\theta) &= \frac{\partial}{\partial\theta}\sum_{i=1}^{m}(h_\theta(x^{i})x_j^i -y^ix^i_j) \\ &= \frac{\partial}{\partial\theta}\sum_{i=1}^{m}(h_\theta(x^{i})x_j^i) \\ &= \frac{\partial}{\partial\theta}\sum_{i=1}^{m}\frac{x^{i}}{1+e^{-z}} \\ &= x^2 h_\theta(x) ^2 \end{align*}

For convenience, define some variables and their differentials \eqalign{ z &= X^T\theta, &\,\,\,\, dz = X^Td\theta \cr p &= \exp(z), &\,\,\,\, dp = p\odot dz \cr h &= \frac{p}{1+p}, &\,\,\,\, dh = (h-h\odot h)\odot dz \,= (H-H^2)\,dz \cr } where
$\,\,\,\,H={\rm Diag}(h)$
$\,\,\,\,\odot$ represents the Hadamard elementwise product
$\,\,\,\,\exp$ is applied elementwise
$\,\,\,\frac{p}{1+p}$ represents elementwise division

The cost function can be written in terms of these variables and the Frobenius inner product (represented by a colon) \eqalign{ J &= -\frac{1}{m}\Big[y:\log(h) + (1-y):\log(1-h)\Big] \cr }

The differential of the cost is \eqalign{ dJ &= -\frac{1}{m}\Big[y:d\log(h) + (1-y):d\log(1-h)\Big] \cr &= -\frac{1}{m}\Big[y:H^{-1}dh - (1-y):(I-H)^{-1}dh\Big] \cr &= -\frac{1}{m}\Big[H^{-1}y - (I-H)^{-1}(1-y)\Big]:dh \cr &= -\frac{1}{m}\Big[H^{-1}y - (I-H)^{-1}(1-y)\Big]:H(I-H)dz \cr &= -\frac{1}{m}\Big[(I-H)y - H(1-y)\Big]:dz \cr &= -\frac{1}{m}(y-h):X^Td\theta \cr &= \frac{1}{m}X(h-y):d\theta \cr } The gradient \eqalign{ G =\frac{\partial J}{\partial\theta} &= \frac{1}{m}X(h-y) \cr } (NB: Your gradient is missing the $\frac{1}{m}$ factor)

The differential of the gradient \eqalign{ dG &= \frac{1}{m}X\,dh \cr &= \frac{1}{m}X(H-H^2)\,dz \cr &= \frac{1}{m}X(H-H^2)X^T\,d\theta \cr\cr } And finally, the gradient of the gradient (aka the Hessian) \eqalign{ \frac{\partial^2J}{\partial\theta\,\partial\theta^T} &= \frac{\partial G}{\partial\theta} = \frac{1}{m}X(H-H^2)X^T \cr }

• by Diag(h) you mean Diagonal Matrix of h ? Aug 10, 2019 at 17:06