Logistic regression with weighted instances

Question

I'm working on implementing a logistic regression algorithm in code. It's based this link. Unfortunately, the paper doesn't talk about weighting the individual examples $x_{i}$.

I think the relevant log likelihood function will look something like this:

$$ L(\vec{w}) = \sum_{i=1}^n \log{g(y_i z_i)} r_i $$

as opposed to what's in the paper:

$$ L(\vec{w}) = \sum_{i=1}^n \log{g(y_i z_i)} $$

where $z_i=\sum_k w_k x_{ik}$ and $r_i$ is the instance weight for the given instance $i$. Also, $y_i \in\{-1,1\}$ in this case, and $g$ is the sigmoid function so $1-g(z)=g(-z)$. This is discussed in the link.

Unfortunately, my math skills are not solid enough to be able to solve for the first and second partial derivatives, which are required to perform the optimization. Without the instance weights I'd like to add, the derivatives are:

$$ \frac{\partial{L}}{\partial{w_{k}}} = \sum_{i=1}^{n}y_{i}x_{ik}g(-y_{i}z_{i}) $$ $$ \frac{\partial^{2}{L}}{\partial{w_{j}}\partial{w_{k}}} = -\sum_{i=1}^{n}x_{ij}x_{ik}g(y_{i}z_{i})g(-y_{i}z_{i}) $$

How do these translate with the new $r_{i}$ instance weight involved? Does this belong on CrossValidated instead? Seemed like maybe because I'm looking for the derivatives it belonged here...

Thanks!

Answer 1

It is worth noting, that for an arbitrary $g(z)$ there is no closed form solution for the optimization problem you mention, even for the case of equal weights. Typically, a gradient-based optimization algorithm is used to obtain a solution (see chapters 2.2 and 2.3 from the paper you used).

For a weighted generalized linear regression you can act in the same way. You only need to calculate derivatives for quasi-likelihood with weights you mentioned and run an optimization routine.

And yes, this question belongs to CrossValidation.