0
$\begingroup$

I was reading about Locally weighted regression in paper written on Locally Weighted Learning by CHRISTOPHER G. ATKESON1, ANDREW W. MOORE and STEFAN SCHAAL that came up in Artificial intelligence review. . But I could not understand it fully. Especially the difference they say between the Distance weighted averaging and local weighted regression.

The equation to find the $ \hat y $ which is the prediction got with respect to the training set given.

In general,

C(q) = $ \sum_{i=0}^n\biggr(\big (\hat y - y_i )^2 \times K(d(x_i, q)) \biggr) $

For Distance weighted averaging,

$$ \hat y = \frac {\sum y_i K(d(x_i, q)} {K(d(x_i, q)} $$

For locally weighted regression,

$$ \hat y = x_i ^ T \beta $$

I am clear that I am missing something while understanding these two as I am not able to understand the physical interpretation diagram they have given,

for distance weighted averaging it is

enter image description here

for Locally weighted averaging it is

enter image description here enter image description here

Can some one explain why the author says in case of locally weighted averaging the string can pull in such a way that the line can translate and rotate and whereas in weighted averaging it can only move up or down ?

$\endgroup$
0
$\begingroup$

Distance weighted averaging is known as kernel smoothing, or the Nadaraya-Watson (NW) estimator, which is a special case of locally weighted regression (LRR). In particular, NW can be understood as the locally weighted regression with 0th-order degree basis (i.e., one single number), while locall weighted regression typicall allows for a polynomial basis.

One important difference between NW and LRR is that the latter allows sharper statistical rate over smoother function class. In particular, NW/kernel smoothing is minimax optimal over Lipschitz functions, while LRR is minimax optimal over Holder class of any order, provided that the correct order is used in LRR. On the other hand, NW is provably sub-optimal for Holder class (or actually any typical function class) with a higher order.

I suggest you read Jianqing Fan's seminal book

https://www.amazon.com/Local-Polynomial-Modelling-Its-Applications/dp/0412983214

to get a deeper understanding of the approach.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.