Meaning of correlation In probability books it is sometimes mentioned that correlation is a measure of linearity of a relationship between random variables. 
This claim is supported by the observation that $\left| \rho(X,Y)\right|=1 \iff X=b+aY$.
But let consider a less extreme case: $X,Y,Z$ are random variables such that $\rho(X,Y)=0.5$ and $\rho(Y,Z)=0.8$. Does it mean that the relationship between $X$ and $Y$ is "less" linear than between $Y$ and $Z$? How at all the degree of linearity in a relationship can be defined (especially given the fact, that a relationship between random variables can be non-functional, so one can not always use the second derivative)?    
Edit:
I am talking about Pearson correlation here.
 A: For ease of exposition, let us consider zero-mean unit-variance jointly continuous random variables $X$ and $Y$ with joint density $f(x,y)$. Then,
$$\begin{align}
\rho = E[XY] &= \int_{-\infty}^\infty \int_{-\infty}^\infty
x\cdot y\cdot f(x,y)\,\mathrm dx\,\mathrm dy\\
&= \int_0^\infty \int_0^\infty xy\big[f(x,y)+f(-x-y)-f(-x,y)-f(x,-y)\big]
\,\mathrm dx\,\mathrm dy.
\end{align}$$
If $\rho > 0$, then the term 
$\big[f(x,y)+f(-x-y)-f(-x,y)-f(x,-y)\big]$ cannot be negative for all
$(x,y)$ in the first quadrant; indeed, roughly speaking,
$$f(x,y)+f(-x-y) > f(-x,y)+f(x,-y)$$
quite often, that is, there is generally more mass in the vicinity of $(x,y)$ and $(-x-y)$ (i.e. in the first and third quadrants) than in the 
vicinity of $(-x,y)$ and $(x,-y)$, (i.e. in the second and fourth quadrants). For $\rho$ close to $1$, very roughly speaking, the probability mass lies in the vicinity of the line $x=y$.
For $\rho < 0$ and for $\rho$ close to $-1$, similar remarks apply
mutatis mutandis. 
The simulations in 
this answer of Michael Hardy illustrate this notion very well. My point is that this sort
of behavior (more mass in two opposite quadrants than in the complementary
opposite quadrants when $|\rho|$ is close to $1$) is a general feature shared by all (zero-mean unit-variance) random variables. Similar
remarks apply when the mean point is not the origin (the quadrants
are defined w.r.t. the mean point) and the variances are different
(the slopes of the lines near which the mass lies are different)
A: It depends on what measure of correlation you use.
The Pearson correlation (the most commonly used one) measures the linearity of a relationship between two random variables.
The Spearman rank correlation however, also measures nonlinear (monotonic) relationships. This is defined as the Pearson correlation coefficient between the ranked variables.
A: Perhaps rather than saying "less linear" you might say "less well modeled by a line". If you think of a scatter-plot of $(x,y)$ pairs, it might be sort of a disk or filled ellipse of points, or an amorphous blob. If it's an ellipse, then you'd say that a line drawn along the major axis of the ellipse "sort of fits" the data, especially if the ellipse is highly eccentric. This is exactly the situation in which the correlation is large. If the scatter plot is amorphous, like the standard drawing of an amoeba, then you'd say that it's hard to fit any particular line to the shape; in this case, correlation tends to be small. 
I don't think that there's anything deeper here than that; the idea is simply that when the correlation is high, a scatter plot of the data tends to look pretty much like a highly eccentric ellipse, which itself looks a  lot like a line.   
A: correlation $=0.99$:

correlation $=0.95$:

correlation $=0.9$:

correlation $=0.7$:

correlation $=0.5$:

correlation $=0.3$:

correlation $=0$:

correlation $=-0.3$:

correlation $=-0.5$:

correlation $=-0.7$:

correlation $=-0.9$:

correlation $=-0.95$:

correlation $=-0.99$:

Appendix: Here is how I created these plots.
First I took a sample from a bivariate normal distribution with correlation $0$.  This of course gave me a scatterplot with non-zero correlation.
Then I replaced each $y$ value with the corresponding residual $y-\hat y$ where $\hat y$ is the fitted value, i.e. the $y$-coordinate of the least-squares line for predicting $y$ based on $x$.  This has two effects: $(1)$ It makes the correlation exactly $0$, and $(2)$ it makes the average of the $y$ values exactly $0$.
Then I divided the $y$ values by the standard deviation of the $y$ values.  I also subtracted the average $x$ value from all of them, and then divided them all by their standard deviation.  At this point I had a scatterplot in which the two averages were $0$, the two standard deviations were $1$, and the correlation was $0$.  That is the plot with correlation $0$ that you see above.
Then, to get a plot with correlation $\rho$, I replaced each data point $\begin{bmatrix} x \\ y \end{bmatrix}$ with
$$
\frac 1 2 \begin{bmatrix} \sqrt{1+\rho}+\sqrt{1-\rho}, & \sqrt{1+\rho}-\sqrt{1-\rho} \\ \sqrt{1+\rho}-\sqrt{1-\rho}, & \sqrt{1+\rho}+\sqrt{1-\rho} \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix}.
$$
That $2\times 2$ matrix is the positive-definite symmetric square root of the correlation matrix $\begin{bmatrix} 1, & \rho \\ \rho, & 1 \end{bmatrix}$.
A: I've added a graphical answer, and I think many books are deficient in this sort of graphics.
Here's another aspect of the meaning of correlation.
For vectors $(x_1,\ldots,x_n),\ (y_1,\ldots,y_n)\in\mathbb R^n$ suppose the coefficients $\hat\alpha,\hat\beta$ are so chosen as to minimize the sum of squares of the residuals $\hat y_i-y_i$ where $\hat y_i$ is the fitted value $\hat y_i=\hat\alpha + \hat\beta x_i$. ("Residuals" should not be confused with "errors".) Let $\bar y = (y_1+\cdots+y_n)/n$.  Then
\begin{align}
\sum_{i=1}^n (y_i-\bar y)^2 & \text{ is the total sum of squares.} \\[6pt]
\sum_{i=1}^n (\hat y_i - \bar y)^2 & \text{ is the explained sum of squares.} \\[6pt]
\sum_{i=1}^n (y_i - \hat y_i)^2 & \text{ is the unexplained sum of squares.}
\end{align}
The total sum of squares it he sum of its explained and unexplained components.  (Note that this statement relies on the orthogonality between the vector of fitted values $\hat y_i$ and the vector of residuals $y_i-\hat y_i$, which the reader may verify.)
Then the square of the correlation is the proportion of the total sum of squares that is explained.
A: Pearson correlation indicates how close the scatter-plot of $(x,y)$ points are to a line. It does not tell you if the relationship between them is linear.
Consider the following graph from Sturdy Econometrics:

Suppose you had planned a regression to estimate the Cuban consumption function and, using income and expenditure data from the Cuban national accounts, you find the estimated equation to be
$$ \mathrm{Expenditure} = 1.9 + 0.69 \times \mathrm{Income} $$
with an $R^2$ of 0.71. This seems like a pretty good equation with a high $R^2$ and a coefficient with the right sign. But suppose further that you had the presence of mind to look at a scatter of observations [as shown below]. The message in these data is very different than the one suggested by the estimated regression. Clearly, the chance that any batch-oriented processing could have picked up this message is very remote indeed. … You would therefore be wise to reserve the right to do some interactive processing and to adjust the way you interpret the data if something unanticipated is discovered.


A: Late, but it is rather for myself.
Let $X,Y$ be random variables. We want to check if one variable can be linearly approximated well enough by another variable. So take the loss function $R(a, b)=E[X-(aY+b)]^2$. We want to find $(a, b)$ such that $R(a, b)$ is minimal. Note that $R(a, b)$ is convex, so a local minimum coincides with the global minimum.
$$\frac{\partial R(a,b)}{\partial a}=-2EXY+2aEY^2+2bEY=0$$
$$\frac{\partial R(a,b)}{\partial b}=-2EX+2aEY+2b=0$$
Solving these equations, we get $a^*=\frac{EXY-EXEY}{EY^2-(EY)^2}=\frac{Cov(X,Y)}{Var(Y)},\ b^*=EX-\frac{Cov(X,Y)}{Var(Y)}EY$. Then after some lengthy calculations we get that
$$R(a^*,b^*)=Var(X)(1-\frac{Cov(X,Y)^2}{Var(X)Var(Y)})=Var(X)(1-r^2)$$
Where $r=\frac{Cov(X,Y)}{\sqrt{Var(X)}\sqrt{Var(Y)}}$ is the correlation coefficient. When $r=\pm1$ (and $r^2\leq1$ by the Cauchy inequality) we have that $0=R(a^*,b^*)=E[X-(a^*Y+b^*)]^2$, so $X=a^*Y+b^*$ almost surely.
Also $r^2>0$ shows some degree of linear dependence as $X$ can be approximated by a linear function of $Y$ with the loss, which is better than the worst possible case $R(a^*,b^*)=Var(X)$.
As $a^*=r\frac{\sqrt{Var(X)}}{\sqrt{Var(Y)}}$, the correlation coefficient shows positive or negative linear dependence.
And from $R(a^*,b^*)=Var(X)(1-r^2)$ it is clear why $r^2$ is "the proportion of the variance in the dependent variable that is predictable from the independent variable(s)". wiki
