Relation between correlation and regression Let $y\in \mathbb{R}$ be a random variable. Let $y$ be expressed as a linear combination of $x_i$ $i=1,2,\cdots,n$, as follows
\begin{equation}
y = \sum\limits_{i=1}^nw_ix_i + \epsilon
\end{equation}
where, $\epsilon$ can be treated as error in regression. Suppose that we solve for the values of $w_i$ using ordinary least squares. Can we say the followings?:


*

*If $\epsilon$ is small for the calculated values of $w_i$, $y$ is strongly correlated with $x_i,$ $i = 1,2,\cdots,n$.

*If $\epsilon$ is large for the calculated values of $w_i$, $y$ is weakly correlated with $x_i,$ $i = 1,2,\cdots,n$

*If one or few of the $x_i$ are linearly dependent upon $x_j$, $j=1,2,\cdots,n$ and $j\ne i$, then removal of dependent $x_i$ from basis functions will improve the regression i.e. $\epsilon$ will decrease. 

 A: You wrote:
$$
y = \sum\limits_{i=1}^n w_ix_i + \varepsilon
$$
with a separate value of $w_i$ for each $x_i$, and just one value of $\varepsilon$ and just one value of $y$.  When you do that, you can get an exact fit, so all residuals are $0$.  Normally one would write something like $y_i = w x_i + \varepsilon_i$ and then estimate $w$ (which is the same for all values of $i$) by least squares.  In that case the equation $y_i = w x_i$ for $i=1,\ldots,n$ has no exact solution for $w$, so one seeks the value of $w$ that minimizes $\sum_{i=1}^n (wx_i - y_i)^2$.  Even more frequently, one has something like $y_i = w_0 + w_1 x_i + \varepsilon_i$ for $i=1,\ldots,n$ and one estimates both $w_0$ and $w_1$.
What is usually called "correlation" is most closely associated with the sort of simple linear regression in which you have
$$
y_i = \sum_{i=1}^n w_0 + w_1 x_i + \varepsilon_i
$$
where you have just as many $y_i$ and $\varepsilon_i$ as $x_i$, and the things to be estimated by least squares, $w_0$ and $w_1$, do not become more numerous as the sample size $n$ increases.
In such a case, the correlation based on the sample $\{(x_i,y_i):i=1,\ldots,n\}$ is
\begin{align}
r & = \frac{\sum_i (y_i - \bar y)(x_i - \bar x)}{\sqrt{\sum_i (x_i-\bar x)^2 \sum_i (y_i -\bar y)^2}} = \frac{\frac 1 n \sum_i (x_i-\bar x)(y_i -\bar y) }{s_x s_y} \\[10pt]
& \text{where } \bar x = \frac 1 n \sum_i x_i \text{ and } \bar y = \frac 1 n \sum_i y_i\\[8pt]
& \text{and } s_x^2 = \frac 1 n \sum_i (x_i-\bar x)^2 \text{ and } s_y^2 = \frac 1 n \sum_i (y_i-\bar y)^2.
\end{align}
Let $\hat w_0$, $\hat w_1$ be the least-squares estimates of $w_0$ and $w_1$; let $\hat y_i = \hat w_0 + \hat w_1 x_i$ be the fitted values; let $\hat\varepsilon_i$ be the residuals (as opposed to the errors $\varepsilon_i$).  Then the least-squares estimates $\hat w_0, \hat w_1$ are the values that minimize $\sum_i \hat\varepsilon_i^2$.
One can show that
$$
\frac{\hat y_i - \bar y}{s_y} = r \frac{x_i-\bar x}{s_x}. \tag 1
$$
That is trivially equivalent to
$$
\hat y_i = \left( r \frac{s_y}{s_x}  \right) x_i - \left( r \frac{s_y}{s_x} \bar x - \bar y \right),  \tag 2
$$
so
$$
\hat w_0 = \bar y - r \frac{s_y}{s_x} \bar x \quad \text{and} \quad \hat w_1 = r \frac{s_y}{s_x}. \tag 3
$$
So $(1)$, $(2)$, and $(3)$ give you a connection between the sample correlation $r$ and the least-squares estimates $\hat w_0$ and $\hat w_1$.
The total sum of squares due of variability of $y_i$ is
$$
\text{SS}_\text{total} = \sum_i (y_i - \bar y)^2. \tag 4
$$
The part of that sum of squares that is explained by variability of $x_i$ is
$$
\text{SS}_\text{explained} = \sum_i (\hat y_i-\bar y)^2. \tag 5
$$
The part that is unexplained is
$$
\text{SS}_\text{unexplained} = \sum_i (y_i - \hat y_i)^2 = \sum_i \hat\varepsilon_i^2. \tag 6
$$
With some algebra you can show that $(4)$ is the sum of $(5)$ and $(6)$.  The proportion of the sum of squares that is explained is then
$$
\frac{\sum_i (\hat y_i - \bar y)^2}{\sum_i (y_i - \bar y)^2}. \tag 7
$$
With some more algebra you can show (the punch line) that $(7)$ is equal to $r^2$.  Thus the square of the correlation is large if the sum of squares of residuals is small compared to the total sum of squares.
