# Find optimal weight vector $\sum_i \log \left(1 + \exp\left\{ -t_i \left(w^T x_i\right)\right\} \right) + \mu \|w \|_2^2$?

Let the function is, \begin{align} f(w) &= \sum_i \log \left(1 + \exp\left\{ -t_i \left(w^T x_i\right)\right\} \right) + \mu \|w \|_2^2 \ , \end{align} where $$t_i \in \mathbb{R}$$, $$w, x_i \in \mathbb{R}^n$$, and $$\mu \in \mathbb{R}$$.

The gradient of the above function is \begin{align} \frac{\partial f}{\partial w} &= \sum_i \left( \frac{-t_i x_i \exp\left\{ -t_i \left(w^T x_i\right)\right\}}{1 + \exp\left\{ -t_i \left(w^T x_i\right)\right\} } \right) + 2\mu w \ \ \\ &= \sum_i \left( \frac{-t_i x_i }{1 + \exp\left\{ +t_i \left(w^T x_i\right)\right\} } \right) + 2\mu w \ . \end{align}

Or equivalently in nice matrix form is--see [1]: \begin{align} \frac{\partial f}{\partial w} = -X \left(I + P \right)^{-1} t + 2\mu w \ , \end{align} where $$X$$ consists of $$\{x_i\}$$ column wise, $$t = [t_1, \ldots, t_n]^T$$ and $$P = {\rm Diag} \left(\exp \left(t \circ X^T w \right)\right)$$. And $${\rm Diag}$$ creates a diagonal matrix.

Question: How to find optimal $$w^*$$?

ADD: This problem is referred as L2 regularized logistic regression (Thanks to Dohamatob for the information). It doesn't have analytical solution. So, please ignore this question.

Partial attempt

I just can't make any progress after setting $$\frac{\partial f}{\partial w} = 0 \Rightarrow 2\mu w = X \left(I + {\rm Diag} \left(\exp \left(t \circ X^T w \right)\right)\right)^{-1} t$$. Can someone help me to find optimal $$w^*$$ analytically?

Thank you

• This problem is called L2-penalized logistic regression. Please always state the context / setup of your problems! BTW, there is no analytical solution. Due to smoothness and strong convexity, accelerated gradient descent schemes should do a great job though. – dohmatob Feb 5 at 13:25
• thank you for your reply, dohmatob. Good to know... – user550103 Feb 5 at 16:03