Is my Gradient and Hessian of the following correct? \begin{align} f &= \sum_i \log \left(1 + \exp\left\{ -t_i \left(w^T x_i\right)\right\} \right) + \mu \|w \|_2^2 \ , \end{align} where $t_i \in \mathbb{R}$, $w, x_i \in \mathbb{R}^n$, and $\mu \in \mathbb{R}$.

I want to find the gradient and Hessian w.r.t. $w$, that is $\frac{\partial f}{\partial w}$ and $\frac{\partial^2 f}{\partial w^2}$.

Partial attempt


\begin{align} \frac{\partial f}{\partial w} &= \sum_i \left( \frac{-t_i x_i \exp\left\{ -t_i \left(w^T x_i\right)\right\}}{1 + \exp\left\{ -t_i \left(w^T x_i\right)\right\} } \right) + 2\mu w \ \ \\ &= \sum_i \left( \frac{-t_i x_i }{1 + \exp\left\{ +t_i \left(w^T x_i\right)\right\} } \right) + 2\mu w \ . \end{align}

is this Gradient correct?


\begin{align} \frac{\partial^2 f}{\partial w^2} &= \frac{\partial}{ \partial w} \left[ \sum_i \left( \frac{-t_i x_i }{1 + \exp\left\{ +t_i \left(w^T x_i\right)\right\} } \right) + 2\mu w \right] \\ &= \sum_i \left( \frac{t_i^2 x_i x_i \ \exp\left\{ +t_i \left(w^T x_i\right)\right\}}{\left(1 + \exp\left\{ +t_i \left(w^T x_i\right)\right\}\right)^2 } \right) + 2\mu I \ . \end{align}

I Think my Hessian is for sure incorrect, isn't it? because I am getting in the numerator of the first part as $x_i x_i$... how would two vectors just multiply :( ...

  • $\begingroup$ The last part should have the outer product $x_i x_i^\top$ instead of $x_i x_i$. $\endgroup$ – angryavian Feb 3 at 18:35
  • $\begingroup$ ok, can you derive that? $\endgroup$ – learning Feb 3 at 18:38

The gradient looks correct, but the hessian doesn't. Here's how I did the calculations.

In order to write the function in purely matrix form, first note that the $\{x_i\}$ vectors are columns of a single matrix $X$. Next use $(\circ)$ to denote the elementwise/Hadamard product and (:) to denote the trace/Frobenius product, i.e. $$A:B = {\rm Tr}(A^TB)$$

Define the following variables. $$\eqalign{ a &= t\circ X^Tw &\implies da = t\circ X^Tdw \cr b &= \exp(-a) &\implies db = -b\circ da \cr p &= \exp(a) &\implies dp = p\circ da \implies 1=b\circ p \cr c &= \log(1+b) &\implies dc = \frac{db}{1+b} \cr }$$ Write the function in terms of these variables. Then calculate its differential and and back-substitute variables until we arrive at the gradient with respect to $w$. $$\eqalign{ f &= \mu\,w:w + 1:c \cr df &= 2\mu\,w:dw + 1:dc \cr &= 2\mu\,w:dw + \frac{1}{1+b}:db \cr &= 2\mu\,w:dw - \frac{1}{1+b}:b\circ da \cr &= 2\mu\,w:dw - \frac{b}{1+b}:t\circ X^Tdw \cr &= 2\mu\,w:dw - X\Big(\frac{t\circ b}{1+b}\Big):dw \cr &= \bigg(2\mu\,w - X\Big(\frac{t}{p+1}\Big)\bigg):dw \cr g = \frac{\partial f}{\partial w} &= 2\mu\,w - X\Big(\frac{t}{1+p}\Big) \cr }$$ Now find the differential and gradient of $g$. $$\eqalign{ dg &= 2\mu\,dw + X\Big(\frac{t\circ dp}{(1+p)\circ(1+p)}\Big) \cr &= 2\mu\,dw + X\Big(\frac{t\circ p\circ da}{1+2p+p\circ p}\Big) \cr &= 2\mu\,dw + X\Big(\frac{t\circ p\circ t\circ X^Tdw}{1+2p+p\circ p}\Big) \cr }$$ Replace the Hadamard products with diagonal matrices, e.g. $$\eqalign{ P &= {\rm Diag}(p),\,\, T &= {\rm Diag}(t),\,\, I &= {\rm Diag}(1) \cr Px &= p\circ x \cr }$$ Therefore $$\eqalign{ dg &= \Big(2\mu I + X(I+2P+P^2)^{-1}T^2PX^T\Big)\,dw \cr H = \frac{\partial g}{\partial w} &= 2\mu I + X(I+2P+P^2)^{-1}T^2PX^T \cr }$$

  • $\begingroup$ great work, greg. I tried to compute myself but I gave up. Now I got curious and tried to compute the optimal weight . But it seems that I can't make any progress in that direction either :( ... math.stackexchange.com/questions/3100947/… $\endgroup$ – user550103 Feb 5 at 9:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.