2
$\begingroup$

My objective function that is to be minimized is as follows:

$f = -\sum_{n=1}^{N}log~p(y_{n}^{a}|x_{n},w) + \sum_{n=1}^{N}(y_{n}^{b}-w^{T}x_{n})^{2} +\lambda\|w\|_1$

The first term models the relationship between data $x$ and labels $y^{a}$ using logistic regression, while the second term models data $x$ and labels $y^{b}$ using linear regression. The third term is to enforce sparsity/feature selection.

My question - I found a number of papers that either show how to solve L1 regularized logistic regression or linear regression. However, I could not find any method that could be applied to both kinds of problems. Do there exist techniques that can help me solve the above problem?

$\endgroup$
2
$\begingroup$

First, a disclaimer: I'm not sure I see the statistical validity of combining both linear and logistic regression with the same measurement vectors $x_n$. I am going to assume you know what you are doing :-) and address the optimization question only.

Some quick and dirty approaches:

  • My Matlab toolbox CVX 2.1 can handle this, although with a caveat because it has to jump through some hoops to get the underlying solvers to accept the logistic regression term.
  • CVX 3.0 beta coupled with the SCS solver can solve this problem "natively", thus avoiding the aforementioned caveat; but this will be a bit more difficult to get up and running, and again, it's a beta!.
  • YALMIP can probably handle this well, too; and I believe it connects to SCS as well, which means it can also solve this problem natively.
  • CVXPY coupled with SCS can do this same thing in Python.
  • And you can implement your own proximal gradient solver if you are so inclined, though of course that's an advanced approach. You'd have to build a function to compute your own derivatives of the smooth portion of the objective.

Here is a logistic regression example for CVX, so you can see how to express the logistic term in a compliant manner using the CVX function log_sum_exp. It's a simple matter to modify this example to add the additional terms.

My recommendation is that you provide weighting values for both the linear regression and $\ell_1$ terms. That is, minimize something like this: $$f = -\sum_{n=1}^{N}\log~p(y_{n}^{a}|x_{n},w) + \lambda_1\sum_{n=1}^{N}(y_{n}^{b}-w^{T}x_{n})^{2} +\lambda_2\|w\|_1$$ You won't know what the best values of $\lambda_1$ and $\lambda_2$ are until you have done some cross validation. What I do know is that the chance that $\lambda_1=1$ is your best choice is slim to none.

The model in CVX is going to look something like this, and assumes that the data $y^a_n$, $y^b_n$ are stored in column vectors ya and yb, respectively, and the vectors the columns of the matrix X.

cvx_begin
    variable w(m)
    minimize(...
        -ya'*X*w+sum(log_sum_exp([zeros(1,m); w'*X'])) ... %logistic
        +lambda1*sum_square(yb-X*w) ... %linear
        +lambda2*norm(w,1)) %regularizer
cvx_end
$\endgroup$
  • $\begingroup$ I wish CVX 3.0 would finish its beta phase one day :-). $\endgroup$ – Royi Aug 28 '17 at 14:51
  • $\begingroup$ Royi, would you do me a favor and stop bugging me about this? I'm sure you mean well, and I would like to see it finished, too. But I have to pay the bills, and developing CVX, unfortunately, does not do that. I will finish it when I have time. $\endgroup$ – Michael Grant Aug 28 '17 at 15:04
  • $\begingroup$ No intention bugging you. It was out of appreciation not trying to bug anyone. Sorry for that. $\endgroup$ – Royi Aug 28 '17 at 15:05
  • $\begingroup$ My apologies for being harsh. But you are a persistent fellow! :-) $\endgroup$ – Michael Grant Aug 28 '17 at 15:06
  • $\begingroup$ Being consistent is a necessary condition for being lucky. Unfortunately it is not sufficient :-). So I keep trying my luck. $\endgroup$ – Royi Aug 28 '17 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.