Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm working on implementing a logistic regression algorithm in code. It's based this link. Unfortunately, the paper doesn't talk about weighting the individual examples $x_{i}$.

I think the relevant log likelihood function will look something like this:

$$ L(\vec{w}) = \sum_{i=1}^n \log{g(y_i z_i)} r_i $$

as opposed to what's in the paper:

$$ L(\vec{w}) = \sum_{i=1}^n \log{g(y_i z_i)} $$

where $z_i=\sum_k w_k x_{ik}$ and $r_i$ is the instance weight for the given instance $i$. Also, $y_i \in\{-1,1\}$ in this case, and $g$ is the sigmoid function so $1-g(z)=g(-z)$. This is discussed in the link.

Unfortunately, my math skills are not solid enough to be able to solve for the first and second partial derivatives, which are required to perform the optimization. Without the instance weights I'd like to add, the derivatives are:

$$ \frac{\partial{L}}{\partial{w_{k}}} = \sum_{i=1}^{n}y_{i}x_{ik}g(-y_{i}z_{i}) $$ $$ \frac{\partial^{2}{L}}{\partial{w_{j}}\partial{w_{k}}} = -\sum_{i=1}^{n}x_{ij}x_{ik}g(y_{i}z_{i})g(-y_{i}z_{i}) $$

How do these translate with the new $r_{i}$ instance weight involved? Does this belong on CrossValidated instead? Seemed like maybe because I'm looking for the derivatives it belonged here...


share|cite|improve this question

It is worth noting, that for an arbitrary $g(z)$ there is no closed form solution for the optimization problem you mention, even for the case of equal weights. Typically, a gradient-based optimization algorithm is used to obtain a solution (see chapters 2.2 and 2.3 from the paper you used).

For a weighted generalized linear regression you can act in the same way. You only need to calculate derivatives for quasi-likelihood with weights you mentioned and run an optimization routine.

And yes, this question belongs to CrossValidation.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.