# Iterative re-weighted least squares derivation

Text states

For a cost function with robust error function $h(e_i)$ $$E(p)=\sum\limits_{i=0}^nh(e_i)$$ it is possible to find an equivalent weighted $L_2$ cost $$E_w(p)=\sum\limits_{i=0}^nw(e_i)\|e_i\|^2,$$ with $$w=\frac{h'(e)}{|e|}$$

In the above, $p$ stands for the parameters of the model we are optimizing. The notation is weird, but I copied it directly from the text I am reading (the error $e_i$ is a function of the parameters and the input).

My question is, how is this derived? Particularly, how is the equality with the weights derived?

The text I am reading is very vague, so I do not know how to go about the derivation. I would appreciate any pointers.

EDIT: My current attempt

Lets say

$$E(x,p,w)=\sum\limits_{i=1}^nw_ie_i(p,x)$$ But we wish to minimize $$E_h(x,p,w)=\sum\limits_{i=1}^nh\circ e_i(p,x)$$ We desire $$\frac{\partial C}{\partial p}=0\iff\frac{\partial C_h}{\partial p}=0$$ Thus $$\frac{\partial }{\partial p}(w_ie_i(x,p))=\frac{\partial }{\partial p}(h\circ e_i(x,p))$$ $$\implies$$ $$w_i\frac{\partial }{\partial p}e_i(x,p)=h'(e_i(x,p))\frac{\partial }{\partial p}e_i(x,p)$$

Thus

$$w_i=h'(e_i(x,p))$$

But this is missing the factor $\frac{1}{|e|}$ the text provides. What am i doing wrong?

• What is the thing you're calling $p$? $\qquad$ – Michael Hardy Dec 11 '16 at 15:07
• I edited the question regarding your comment, and provided my derivation, in hopes for a correction. – LeastSquaresWonderer Dec 11 '16 at 15:28
• So $e_i$ is the $i$th residual when the vector of parameters is $p$? – Michael Hardy Dec 11 '16 at 15:53
• The notation $w(e_i)$ suggested the weight depends on the value of the residual $e_i$, where I would have guessed that what it depends on is the index $i$. $\qquad$ – Michael Hardy Dec 11 '16 at 15:55
• Indeed. I copied the notation from the text I was reading. In my derivation, I employed the notation you advised. Is my derivation correct? – LeastSquaresWonderer Dec 11 '16 at 18:06