# Why are randomly drawn vectors nearly perpendicular in high dimensions

I am struggling understanding this finding. Can somebody explain intuitively why randomly drawn high-dimensional vectors will tend to be mutually orthogonal? I realize that intuition in high dimensions might be too much to ask for, still, an explanation without having to integrate over several pages of symbols would be preferred.

• (A) what is the source for this (B) what do you know so far? Oct 28, 2014 at 21:07
• Others are using more rigour, but I think it simply as follows. By rotational symmetry of the distribution you might as well look at the inner product of a random vector and $(1,0,0,\ldots,0)$. That inner product is zero-mean, but its variance will be $1/n$. So when $n$ is in the hundreds, you need quite a few SDs off the mean to have a significant inner product (for a suitable value of "significant"). Oct 28, 2014 at 22:03
• @JyrkiLahtonen I will edit my answer to incorporate this line of reasoning explicitly. Oct 29, 2014 at 8:33
• @JyrkiLahtonen thank you - this is very helpful. Oct 29, 2014 at 14:56

A random uniform unit vector is $X/\|X\|$ where $X$ is standard normal, thus the scalar product of two independent unit vectors $U$ and $V$ is $\langle U,V\rangle=\langle X,Y\rangle/(\|X\|\cdot\|Y\|)$ where $X$ and $Y$ are independent and standard normal. When $n\to\infty$, by the law of large numbers, $\|X\|/\sqrt{n}\to1$ almost surely and $\|Y\|/\sqrt{n}\to1$ almost surely, and by the central limit theorem, $\langle X,Y\rangle/\sqrt{n}$ converges in distribution to a standard one-dimensional normal random variable $Z$.

Thus, $\sqrt{n}\cdot\langle U,V\rangle\to Z$ in distribution, in particular, for every $\varepsilon\gt0$, $P(|\langle U,V\rangle|\geqslant\varepsilon)\to0$. In this sense, when $n\to\infty$, the probability that $U$ and $V$ are nearly orthogonal goes to $1$.

Likewise, $k$ independent uniform unit vectors are nearly orthogonal with very high probability when $n\to\infty$, for every fixed $k$.

• Concerning your last remark, and interesting variation would be to study the maximum inner product between any two of the $k$ random iid uniform vectors, in the limit when both $k$ and $n$ go to infinity while being related in some sense, say $n^{\ell}\log n \ll k \lesssim n^{\ell+1-\delta}$, for a fixed integer $\ell$ and fixed $\delta > 0$. Apr 11, 2021 at 11:23
• What does "independent uniform unit vectors" mean? Also, not sure what "where $X$ is standard normal" means here either :/ Dec 30, 2021 at 13:16
• Found this page after Google search on similar question. This answer is interesting but it assumes that the vectors are unit vectors, whereas OP's question made no such restriction. Turns out that, in the case where the random vectors are not unit vectors (i.e. are truly random), then they're still likely to be orthogonal, but increasingly less so as the number of dimensions increases. Jan 24, 2022 at 23:23
• Does anyone know of a result showing the rate of convergence toward orthogonality as $n$ grows? e.g. I think a finite sample high probability bound is possible, like $|\langle U,V\rangle| \leqslant \mathcal{O}\left(\log\left(\frac{1}{\delta}\right) \frac{1}{n}\right)$ with probability $1 - \delta$ for any $\delta \in (0, 1)$. Mar 21 at 16:10
• And one might be able to make a more general result which includes a possible nonzero covariance between $U$ and $V$ 🤔 Mar 21 at 16:10

Here is one way to reason, chosen for simplicity of calculations: Consider the unit vector $e=(1,0,0,\ldots,0)\in\mathbb R^n$. One way to measure how 'orthogonal' $e$ is to other vectors is to calculate the average of $(e\cdot x)^2$ as $x$ ranges over the unit sphere. If $S$ denotes the surface measure on the unit sphere corresponding to (normalized) area, then $$\int |e\cdot y|^2 dS(y) =\int |y_1|^2 dS(y)=\frac{1}{n}\int \sum_{j=1}^n |y_j|^2 dS(y)=\frac{1}{n}.$$ Thus, in this sense, vectors are generally 'more' orthogonal in higher dimensional spaces.

Edit: This line of reasoning follows closely the argument given by JyrkiLahtonen in the comments above, as one sees by considering a random $\mathbb R^n$-valued vector $Y$, uniformly distributed on the unit sphere. If we consider the random variable $e\cdot Y$, then $$E \; e\cdot Y=\int e\cdot y \;dS(y)=0,$$ because $S$ in invariant under the transformation $y\mapsto -y$. On the other hand $$V(e\cdot Y)=\int |e\cdot y|^2 dS(y) =\frac{1}{n},$$ as shown above. Therefore, intuitively, $e\cdot Y$ is small when $n$ is large. Rigorously, we can employ Chebyshev's inequality to obtain $$P(|e\cdot Y|\geq \epsilon)\leq \frac{1}{n\epsilon^2}.$$

• In the first computation, it helped me to add some steps. First, note that $\sum_{j=1}^{n}|y_j|^2 = |y|^2 = 1$ when $y \in S$, the unit sphere. In going from $\frac{1}{n}\int n |y_1|^2 dS(y)$ to $\frac{1}{n} \int \sum_{j=1}^{n}|y_{j}|^2 dS(y)$ we might note that we are splitting up the integral and applying dS-invatiant permutations $1\rightarrow j$ for all $j$. V is measuring the variance, and this being small means the average (expectation value) of $e \cdot y$ should be close to 0. We could also apply a dS-invariant rotation matrix to $e$ to show invariance of choice of reference vector. Sep 16, 2020 at 22:09
• Can you explain how you obtained that Chebyshev result? Aug 26 at 4:24

In statistics orthogonality is deeply linked to correlation. Random vectors are orthogonal when the values they contain are uncorrelated, meaning that in the $$x_i$$ values of $$\vec{v_1}$$ and $$\vec{v_2}$$ do not appear common patterns. The higher the dimension is the higher is $$i$$ and so the probability is higher that $$x_{i}$$ value of $$\vec{v_1}$$ and the $$x_i$$ values of $$\vec{v_2}$$ have no common pattern, because the higher the dimension the higher the possibilities (in an urn problem increase the number of balls ...). Equivalent to the statement that $$\vec{v_1}$$ and $$\vec{v_2}$$ have no common patterns is the statement that they are uncorrelated what is equivalent to the statement that they are orthogonal.

So the key to understand why randomly drawn vectors are perpendicular in high dimensions is the deep link between orthogonality and statistical correlation :

https://stats.stackexchange.com/questions/171324/what-is-the-relationship-between-orthogonal-correlation-and-independence

This is linked to a more philosophical question if real randomness actually exists. There are strong hints that this is not the case, i.e. nonlinear dynamical systems (chaotic systems) are able to generate white noise in a deterministic way : https://ieeexplore.ieee.org/document/1236865 - Chaotic maps generating white noise

Here's an excerpt from Lecture 2 of Keith Ball's An Elementary Introduction to Modern Convex Geometry: See the link for details, but as the image shows, the measure of a spherical cap cut off by a hyperplane which is $$\varepsilon$$ away from the origin is bounded above by the ratio between volume of a ball of radius $$\sqrt{1-\varepsilon^2}$$ and the volume of a ball of radius 1, that is, $$(1-\varepsilon^2)^{n/2}$$. So if $$X$$ and $$Y$$ are independent random unit vectors (uniform on the sphere), then the portion of the cap area is $$P(\langle X,Y\rangle > \varepsilon) \le (1-\varepsilon^2)^{n/2} \le e^{-n\varepsilon^2/2}$$ which is close to $$1$$ when $$n$$ is large (if $$\varepsilon$$ is fixed, at least).

For intuition let us reframe asking why some vector is orthogonal to most others as, why is some random vector almost orthogonal to most standard basis vectors?

Now the unit vector which is in some sense least orthogonal to every basis vector is

$$\tfrac1{\sqrt{d}}(1, \dots, 1).$$

Notice how we have to make this vector more orthogonal in some basis $$e_i$$ if we want to make it less orthogonal for some other basis $$e_j$$. Therefore the total "orthogonalness" stays the same. So if we do not concentrate our values in some dimensions, all of them will decrease at rate roughly $$\frac1{\sqrt{d}}$$ with the dimension. And this is the result proven by the accepted answer