1
$\begingroup$

Let $y\in \mathbb{R}$ be a random variable. Let $y$ be expressed as a linear combination of $x_i$ $i=1,2,\cdots,n$, as follows \begin{equation} y = \sum\limits_{i=1}^nw_ix_i + \epsilon \end{equation} where, $\epsilon$ can be treated as error in regression. Suppose that we solve for the values of $w_i$ using ordinary least squares. Can we say the followings?:

  1. If $\epsilon$ is small for the calculated values of $w_i$, $y$ is strongly correlated with $x_i,$ $i = 1,2,\cdots,n$.
  2. If $\epsilon$ is large for the calculated values of $w_i$, $y$ is weakly correlated with $x_i,$ $i = 1,2,\cdots,n$
  3. If one or few of the $x_i$ are linearly dependent upon $x_j$, $j=1,2,\cdots,n$ and $j\ne i$, then removal of dependent $x_i$ from basis functions will improve the regression i.e. $\epsilon$ will decrease.
$\endgroup$
3
  • $\begingroup$ What are your thoughts on the problem? A quick hint for 1: google search "spurious correlations" $\endgroup$
    – TomGrubb
    Commented Jan 4, 2016 at 0:01
  • $\begingroup$ Spurious correlations are related to the correlation between ratio of random variables. But in my case, there is no ratio. $y$ is a linear combination of $x_i$. $\endgroup$
    – user146290
    Commented Jan 4, 2016 at 0:18
  • $\begingroup$ If you have a separate value of $w$ for each $x$, then you can get an exact fit, so all residuals are $0$. Normally one would write something like $y_i = w x_i + \varepsilon_i$ and then estimate $w$ (which is the same for all values of $i$) by least squares. In that case the equation $y_i = w x_i$ for $i=1,\ldots,n$ has no exact solution for $w$, so one seeks the value of $w$ that minimizes $\sum_{i=1}^n (wx_i - y_i)^2$. Even more frequently, one has something like $y_i = w_0 + w_1 x_i + \varepsilon_i$ for $i=1,\ldots,n$ and one estimates both $w_0$ and $w_1$. ${}\qquad{}$ $\endgroup$ Commented Jan 4, 2016 at 0:22

1 Answer 1

3
$\begingroup$

You wrote: $$ y = \sum\limits_{i=1}^n w_ix_i + \varepsilon $$ with a separate value of $w_i$ for each $x_i$, and just one value of $\varepsilon$ and just one value of $y$. When you do that, you can get an exact fit, so all residuals are $0$. Normally one would write something like $y_i = w x_i + \varepsilon_i$ and then estimate $w$ (which is the same for all values of $i$) by least squares. In that case the equation $y_i = w x_i$ for $i=1,\ldots,n$ has no exact solution for $w$, so one seeks the value of $w$ that minimizes $\sum_{i=1}^n (wx_i - y_i)^2$. Even more frequently, one has something like $y_i = w_0 + w_1 x_i + \varepsilon_i$ for $i=1,\ldots,n$ and one estimates both $w_0$ and $w_1$.

What is usually called "correlation" is most closely associated with the sort of simple linear regression in which you have $$ y_i = \sum_{i=1}^n w_0 + w_1 x_i + \varepsilon_i $$ where you have just as many $y_i$ and $\varepsilon_i$ as $x_i$, and the things to be estimated by least squares, $w_0$ and $w_1$, do not become more numerous as the sample size $n$ increases.

In such a case, the correlation based on the sample $\{(x_i,y_i):i=1,\ldots,n\}$ is \begin{align} r & = \frac{\sum_i (y_i - \bar y)(x_i - \bar x)}{\sqrt{\sum_i (x_i-\bar x)^2 \sum_i (y_i -\bar y)^2}} = \frac{\frac 1 n \sum_i (x_i-\bar x)(y_i -\bar y) }{s_x s_y} \\[10pt] & \text{where } \bar x = \frac 1 n \sum_i x_i \text{ and } \bar y = \frac 1 n \sum_i y_i\\[8pt] & \text{and } s_x^2 = \frac 1 n \sum_i (x_i-\bar x)^2 \text{ and } s_y^2 = \frac 1 n \sum_i (y_i-\bar y)^2. \end{align}

Let $\hat w_0$, $\hat w_1$ be the least-squares estimates of $w_0$ and $w_1$; let $\hat y_i = \hat w_0 + \hat w_1 x_i$ be the fitted values; let $\hat\varepsilon_i$ be the residuals (as opposed to the errors $\varepsilon_i$). Then the least-squares estimates $\hat w_0, \hat w_1$ are the values that minimize $\sum_i \hat\varepsilon_i^2$.

One can show that $$ \frac{\hat y_i - \bar y}{s_y} = r \frac{x_i-\bar x}{s_x}. \tag 1 $$ That is trivially equivalent to $$ \hat y_i = \left( r \frac{s_y}{s_x} \right) x_i - \left( r \frac{s_y}{s_x} \bar x - \bar y \right), \tag 2 $$ so $$ \hat w_0 = \bar y - r \frac{s_y}{s_x} \bar x \quad \text{and} \quad \hat w_1 = r \frac{s_y}{s_x}. \tag 3 $$ So $(1)$, $(2)$, and $(3)$ give you a connection between the sample correlation $r$ and the least-squares estimates $\hat w_0$ and $\hat w_1$.

The total sum of squares due of variability of $y_i$ is $$ \text{SS}_\text{total} = \sum_i (y_i - \bar y)^2. \tag 4 $$ The part of that sum of squares that is explained by variability of $x_i$ is $$ \text{SS}_\text{explained} = \sum_i (\hat y_i-\bar y)^2. \tag 5 $$ The part that is unexplained is $$ \text{SS}_\text{unexplained} = \sum_i (y_i - \hat y_i)^2 = \sum_i \hat\varepsilon_i^2. \tag 6 $$ With some algebra you can show that $(4)$ is the sum of $(5)$ and $(6)$. The proportion of the sum of squares that is explained is then $$ \frac{\sum_i (\hat y_i - \bar y)^2}{\sum_i (y_i - \bar y)^2}. \tag 7 $$ With some more algebra you can show (the punch line) that $(7)$ is equal to $r^2$. Thus the square of the correlation is large if the sum of squares of residuals is small compared to the total sum of squares.

$\endgroup$
2
  • $\begingroup$ Hi, just to say that minimizing the squared error is equivalent to maximize the likelihood of $\epsilon$ where $\epsilon$ is supposed to follow a normal distribution (maximizing the likelihood being equivalent in that case to minimize its variance) $\endgroup$
    – reuns
    Commented Jan 4, 2016 at 1:57
  • $\begingroup$ @user1952009 : To get that conclusion one should assume the expected value of the normal distribution of the errors is $0$ and the errors are independent. However, maximum likelihood seems perhaps off topic for this question. (Although some uncertainty remains about what the actual question was.) ${}\qquad{}$ $\endgroup$ Commented Jan 4, 2016 at 2:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .