Let the function is, \begin{align} f(w) &= \sum_i \log \left(1 + \exp\left\{ -t_i \left(w^T x_i\right)\right\} \right) + \mu \|w \|_2^2 \ , \end{align} where $t_i \in \mathbb{R}$, $w, x_i \in \mathbb{R}^n$, and $\mu \in \mathbb{R}$.

The gradient of the above function is \begin{align} \frac{\partial f}{\partial w} &= \sum_i \left( \frac{-t_i x_i \exp\left\{ -t_i \left(w^T x_i\right)\right\}}{1 + \exp\left\{ -t_i \left(w^T x_i\right)\right\} } \right) + 2\mu w \ \ \\ &= \sum_i \left( \frac{-t_i x_i }{1 + \exp\left\{ +t_i \left(w^T x_i\right)\right\} } \right) + 2\mu w \ . \end{align}

Or equivalently in nice matrix form is--see [1]: \begin{align} \frac{\partial f}{\partial w} = -X \left(I + P \right)^{-1} t + 2\mu w \ , \end{align} where $X$ consists of $\{x_i\}$ column wise, $t = [t_1, \ldots, t_n]^T$ and $P = {\rm Diag} \left(\exp \left(t \circ X^T w \right)\right)$. And ${\rm Diag}$ creates a diagonal matrix.

Question: How to find optimal $w^*$?

ADD: This problem is referred as L2 regularized logistic regression (Thanks to Dohamatob for the information). It doesn't have analytical solution. So, please ignore this question.

Partial attempt

I just can't make any progress after setting $\frac{\partial f}{\partial w} = 0 \Rightarrow 2\mu w = X \left(I + {\rm Diag} \left(\exp \left(t \circ X^T w \right)\right)\right)^{-1} t$. Can someone help me to find optimal $w^*$ analytically?

Thank you

  • 1
    $\begingroup$ This problem is called L2-penalized logistic regression. Please always state the context / setup of your problems! BTW, there is no analytical solution. Due to smoothness and strong convexity, accelerated gradient descent schemes should do a great job though. $\endgroup$ – dohmatob Feb 5 at 13:25
  • $\begingroup$ thank you for your reply, dohmatob. Good to know... $\endgroup$ – user550103 Feb 5 at 16:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.