# Intuition for Formal Definition of Linear Independence

I learned about this a long time ago but it never really clicked, which led me to these questions:

1. How the formal definition (at the bottom) works. I have a rough intuition: linear independence is where the variables are independent and don't affect each other. But I don't follow the formal definition. I would like to have a deep understanding of the formal definition based on these linear combination equations. I'm not sure how a linear combination constructed in a certain way can tell you the variables are independent or not.
2. Why set the linear combination equations to $\vec{0}$. I don't see how setting to zero helps determine independence or not.
3. Why choose $a_i$ to be non-zero in one case. It seems arbitrary.

From Wikipedia:

A subset $S=\{{\vec {v}}_{1},{\vec {v}}_{2},\dots ,{\vec {v}}_{n}\}$ of a vector space $V$ is linearly dependent if there exist a finite number of distinct vectors ${\vec {v}}_{1},{\vec {v}}_{2},\dots ,{\vec {v}}_{k}$ in $S$ and scalars $a_{1},a_{2},\dots ,a_{k}$, not all zero, such that

$$a_{1}{\vec {v}}_{1}+a_{2}{\vec {v}}_{2}+\cdots +a_{k}{\vec {v}}_{k}={\vec {0}}$$

where ${\vec {0}}$ denotes the zero vector.

The vectors in a set $T=\{{\vec {v}}_{1},{\vec {v}}_{2},\dots ,{\vec {v}}_{n}\}$ are linearly independent if the equation

$$a_{1}{\vec {v}}_{1}+a_{2}{\vec {v}}_{2}+\cdots +a_{n}{\vec {v}}_{n}={\vec {0}}$$

can only be satisfied by $a_{i}=0$ for $i=1,\dots ,n$.

So my understanding is, there are two subsets $S$ and $T$ of $V$. In one of them, the coefficients are not all zero, in the other they are all zero. In one case they are linearly dependent, in the other not. I don't understand why though; that's as much as I understand. Not sure why the equations were constructed like this in the first place.

• Do you have any idea of what would be an intuitive definition of linear dependence or linear independence? What ideas do just the phrases linear (in-)dependence give you? May 13, 2018 at 21:17
• 1. The coefficients are not in the subsets. 2. There are more than two possible subsets, they've just picked one that is linearly independent and one that isn't. 3. This is a definition of linear independence. May 14, 2018 at 1:26
• @immibis: Of course, there is such a thing as the set of all linearly independent subsets of V. This is non-empty whenever V is not empty. (all subsets of V containing only a single vector are linearly independent) May 14, 2018 at 16:12

Imagine you have a collection of arrows pointing in various directions. If they're linearly dependent, then you can stretch, shrink, and reverse (but not rotate) them in such a way that if you lay them head-to-tail then they form a closed loop. For example, if you have three arrows that happen to all lie in the same plane (linearly dependent), then you can form a triangle out of them, but you can't if one of them sticks out of the plane formed by the other two (linearly independent).

• This doesn't help me :/. Wondering how this applies to the equations. May 28, 2018 at 1:02

Before we grapple with linear independence, it might be best to figure out what linear dependence means first.

When I think of the phrase "linear dependence" with regards to a set of vectors, what comes to mind for me is that, in some sense, one of those vectors "depends" on the other vectors in the set. The mathematical formalization for this dependence is:

A set of nonzero vectors $$\{\mathbf{v}_k \}_{k=1}^n$$ in a vector space $$V$$ over a field $$F$$ is called linearly dependent when there exists$$^\dagger$$ a $$j$$ so that one can write $$\displaystyle \mathbf{v}_j = \sum_{k \neq j} c_k \mathbf{v}_k$$, where $$c_k \in F$$.

So we say $$\mathbf{v}_j$$ "depends" on the other vectors: for any $$d \in \mathbb{R}$$, one can arrive at the point $$d\mathbf{v}_j$$ simply by travelling some distance in each of the other directions$$^\ddagger$$ $$\mathbf{v}_{k \neq j}$$. In other words, the span of the entire set of vectors is the same as the span of that same set of vectors but with $$\mathbf{v}_j$$ excluded. Note that, because $$\mathbf{v}_j$$ is nonzero, we must have $$c_k \neq 0$$ for at least one $$k$$.

Rewriting the above as $$\mathbf{v}_j - \displaystyle \sum_{k \neq j} c_k \mathbf{v}_k = 0$$, we can now deduce that a set of $$n$$ vectors is linearly dependent $$\iff$$ we can find a set of constants $$\{c_k\}_{k=1}^n$$, not all zero, so that $$\displaystyle \sum_{k=1}^n c_k \mathbf{v}_k = 0$$. This statement is equivalent to our definition for linear independence:

A set of vectors $$\{\mathbf{v}_k \}$$ is called linearly independent $$\iff$$ we cannot find a set of constants $$\{c_k\}_{k=1}^n$$, not all zero, so that $$\displaystyle \sum_{k=1}^n c_k \mathbf{v}_k = 0$$. This is to say, $$c_k = 0$$ for all $$k$$ is the only solution to this equation.

$$^\dagger$$ This $$j$$ is never unique.

$$^\ddagger$$ To give a concrete example, let $$\displaystyle \mathbf{v}_1 = \left[1 \atop 0 \right], \mathbf{v}_2 = \left[0 \atop 1\right]$$, and $$\displaystyle \mathbf{v}_3 = \left[1 \atop 1\right]$$ in $$\mathbb{R}^2$$. See that $$\mathbf{v}_3$$ depends on $$\mathbf{v}_1$$ and $$\mathbf{v}_2$$ since you can get to the point $$d \mathbf{v}_3$$ for any $$d \in \mathbb{R}$$ by first travelling $$d$$ units in the $$\mathbf{v}_1$$ direction and then $$d$$ units in the $$\mathbf{v}_2$$ direction.

• Having difficulty parsing the "for any $d\in \mathbb{R}$ ... This statement is equivalent...", wondering if you might be able to rephrase or simplify. I'm hoping to understand why the formal definition of linear independence/dependence works / how it tells that the vectors $\textbf{v}_n$ are correlated. Would also like to understand it independent of the numbers/matrices, so a vector space of non-numbers. May 20, 2018 at 18:43

You have linear dependence when a vector can be expressed as a linear combination of the others in the set. Dependence in this sense is like the $y$ in $y=f(x)$, meaning that the $y$ depends on the values of $x$ and is determined by it through $f$. This happens when you can write a linear combination of the vectors that equals to zero, since you can move one of the vectors on the other side and divide by $-a_{i}$. Of course the coefficient must not be all zeroes otherwise you would just have $0=0$.

• "Of course the coefficient must not be all zeroes otherwise you would just have 0=0.". Okay that makes sense, nice totally missed that lol. May 20, 2018 at 18:46
• "when you can write a linear combination of the vectors that equals to zero" still don't see why this determines they should be linearly independent or not, though I understand moving vectors to the other side etc. would help solve the equation. May 20, 2018 at 18:47
• @LancePollard for example say we can write $a_{1}{\vec {v}}_{1}+a_{2}{\vec {v}}_{2}+\cdots +a_{5}{\vec {v}}_{5}={\vec {v}}_{6}$ for non-zero coefficients $a_k$, this means that $a_{1}{\vec {v}}_{1}+a_{2}{\vec {v}}_{2}+\cdots +a_{5}{\vec {v}}_{5}-{\vec {v}}_{6}={\vec {0}}$ for non-zero coefficients $a_k$. And this is true for any possible linear combination. A nice analogy is how I like to think about how $(x-x_1)(x-x_2)(x-x_3)=0$ iff $x=x_1$, $x=x_2$, or $x=x_3$. The former is pretty much a more concise way of encapsulating the logic. Jun 10, 2018 at 19:40

Here's my intuition: Think of vectors as the axes that we use to define a two-dimensional, three-dimensional, or $n$-dimensional space. A set of vectors is linearly dependent when one of the vectors isn't necessary — it doesn't add anything useful to our coordinate system. This happens when one vector is just a linear combination of other vectors in the set.

As an example, consider these three vectors in two-dimensional space:

$\vec w = \begin{bmatrix}2 \\3 \end{bmatrix}$, $\vec x = \begin{bmatrix}1 \\0 \end{bmatrix}$, $\vec y = \begin{bmatrix}0 \\1 \end{bmatrix}$

In this case, we can express $\vec w$ as a linear combination of the other two vectors, $\vec w = 2 \vec x + 3 \vec y$:

$\begin{bmatrix}2 \\3 \end{bmatrix} = 2\cdot \begin{bmatrix}1 \\0 \end{bmatrix} + 3\cdot \begin{bmatrix}0 \\1 \end{bmatrix}$

Equivalently, we can rewrite this as $2\vec x + 3 \vec y - 1\vec w = 0$:

$2\cdot \begin{bmatrix}1 \\0 \end{bmatrix} + 3\cdot \begin{bmatrix}0 \\1 \end{bmatrix} - 1 \begin{bmatrix}2 \\3 \end{bmatrix} = \begin{bmatrix}0 \\0 \end{bmatrix}$

Intuitively, this is what is meant by the proof. When these factors hold, it means that at least one vector is essentially "redundant" and unnecessary — it's just a combination of the other vectors.

More generally, assume you have a set of vectors with this property:

$a_1 \vec v_1 + a_2 \vec v_2 + ... + a_n \vec v_n = 0$

This can be written as:

$-a_1 \vec v_1 = a_2 \vec v_2 + ... + a_n \vec v_n$

Which means that $v_1$ can be expressed as simply a linear combination of the other vectors:

$v_1 = a_2' v_2 + ... + a_n'v_n$

When you have a set of vectors like this, it means that you have multiple ways to express the same point using your coordinate system. For example, with the vectors above, a point could be identified in the standard x/y coordinate system as (8, 15). But the same point could also be expressed as (4, 3) in w/y coordinates or as (5, -2) in w/x coordinates:

$8\vec x + 15 \vec y = 8\begin{bmatrix}1 \\0 \end{bmatrix} + 15 \begin{bmatrix}0 \\1 \end{bmatrix} = \begin{bmatrix}8 \\15 \end{bmatrix}$

$4\vec w + 3 \vec y = 4\begin{bmatrix}2 \\3 \end{bmatrix} + 3\begin{bmatrix}0 \\1 \end{bmatrix} = \begin{bmatrix}8 \\15 \end{bmatrix}$

$5\vec w -2 \vec x = 5\begin{bmatrix}2 \\3 \end{bmatrix} -2\begin{bmatrix}1 \\0 \end{bmatrix} = \begin{bmatrix}8 \\15 \end{bmatrix}$

We'd normally rather have a minimal set of vectors so that there's only one way to represent each point. It keeps the math simpler. This is similar to a system of equations in which you have three equations for two unknowns. In the above, since we have $\vec w = 2 \vec x + 3 \vec y$, we can always substitute $\vec x$ and $\vec y$ for $\vec w$.

Note that there's no "correct" coordinate system -- any set of two vectors here, $\{\vec x, \vec y\}$, $\{\vec w, \vec y\}$, or $\{\vec w, \vec x\}$, would provide a valid basis for a two-dimensional coordinate system. But there's no need to have three vectors here to represent a point in two-dimensional space.

• "A set of vectors is linearly dependent when one of the vectors isn't necessary -- it doesn't add anything useful to our coordinate system." This needs to be in bold or header format. May 14, 2018 at 18:07
• Thanks! Good suggestion. (I'm new here.) May 15, 2018 at 6:35
• By "coordinate system" wondering if you mean basis. May 20, 2018 at 18:48
• This is the closest so far to making sense, thank you so much! Will have to think about this part, alllllmost there :) "This can be written as: $-a_1 \vec v_1 = a_2 \vec v_2 + ... + a_n \vec v_n$" May 20, 2018 at 18:55
• Wondering if you could summarize your examples at the end, I think I just am having difficulty then connecting the $-a_1 \vec v_1 = a_2 \vec v_2 + ... + a_n \vec v_n$ statement and "the same point could also be expressed as (4, 3) in w/y coordinates or as (5, -2) in w/x coordinates" to what this means in terms of linear independence. May 20, 2018 at 18:58

How the formal definition works... I'm not sure how a linear combination constructed in a certain way can tell you the variables are independent or not.

Perhaps a more intutive definition of linear dependence is as follows:

A set of vectors is linearly dependent if one of its vectors is a linear combination of the other vectors.

In other words, one of its vectors "depends linearly" on the other vectors. This definition can be deduced from the formal definition:

If $\{ \mathbf{v}_1, \dots, \mathbf{v}_n \}$ is linearly dependent, then there exists some non-zero $\alpha_1, \dots, \alpha_n$ such that

$$\alpha_1 \mathbf{v}_1 + \alpha_2 \mathbf{v}_2 + \dots + \alpha_n \mathbf{v}_n = \mathbf{0}$$

Without loss of generality, suppose $\alpha_1 \neq 0$ then we have

$$\mathbf{v}_1 = - \frac{\alpha_2}{\alpha_1} \mathbf{v}_2 - \dots - \frac{\alpha_n}{\alpha_1} \mathbf{v}_n$$

So $\mathbf{v}_1$ is a linear combination of the other vectors.

Why set the linear combination equations to $\mathbf{0}$. I don't see how setting to zero helps determine independence or not.

In the proof above, we did not set the right-hand side to zero. It is zero only because everything is moved to the left-hand side.

Why choose $\alpha_k$ to be non-zero in one case. It seems arbitrary.

In the proof above, we need not work with $\alpha_1$. We could have chosen any non-zero $\alpha_k$ to show that $\mathbf{v}_k$ is a linear combination of the other vectors.

• I think this is the best answer in the sense that it gets at the heart of the confusion of the OP, who seems to not have read quite as actively as he should have. A little messing around with the definition should show anyone where the definition comes from. May 15, 2018 at 7:26

If we are given two nonparallel vectors $u,v$ in the space, they span a plane, a 2d subspace. Any vector on that plane can be expressed as $\lambda u+\mu v$, these will be the dependent vectors for $u,v$, everybody else will be independent from $u,v$.
And, by the way, $\lambda,\mu$ are the coordinates w.r.t. basis $u,v$ of plane $\mathrm{span}(u,v)$.

The geometrical meaning of $n$ vectors being linearly independent is that they span exactly $n$ dimensions,
alternatively, they form a basis for the subspace they generate.

A main feature of vector spaces is that we can set up coordinate systems and coordinate all vectors in a unique way.

Setting up a coordinate system is equivalent to give basis vectors.
A system of vectors is good for coordinating all vectors iff it is a generating system, i.e. generates the whole vector space (by means of linear combinations).
A system of vectors $v_k$ is good for coordinating vectors in a unique way iff it is linearly independent.

• Indeed, if $\lambda_1v_1+\dots+\lambda_nv_n=\mu_1v_1+\dots+\mu_nv_n$ are two different coordinates for the same vectors, then $(\lambda_1-\mu_1)v_1+\dots+(\lambda_n-\mu_n)v_n=0$ with at least 1 nonzero coefficient.
• If, on the other hand, $v_i$ uniquely coordinates the vectors, in particular it does so for $0$.
• "Setting up a coordinate system is equivalent to give basis vectors." Nice, that is helpful. May 20, 2018 at 19:00
• "A system of vectors $v_k$ is good for coordinating vectors in a unique way iff it is linearly independent." Wondering what this means in terms of the generating system. If generating system (i.e. a set of vectors) coordinating all vectors in the vector space means that (a) they are basis vectors and (b) they are linear independent. (and, are the basis vectors linear independent/dependent on each other or if it doesn't matter). May 20, 2018 at 19:04
• An $n$ dimensional space is generated by at least $n$ vectors. $n$ will suffice if they are linearly independent, on the other hand any $n+1$ vectors are linearly dependent. May 20, 2018 at 22:52

Your basic intuition is quite right --- a set of vectors are linearly independent if they don't affect each other (which, as it stands, is a somewhat ambiguous statement, hence the formal definition).

Perhaps it would be helpful to start with just two vectors, say the standard vectors $$\begin{pmatrix} 1\\ 0 \end{pmatrix} \hspace{20pt}\text{and}\hspace{20pt} \begin{pmatrix} 0\\ 1 \end{pmatrix}$$ which are linearly independent in $\mathbb{R}^{2}$ (two-dimensional coordinate space). Natural intuition would be to think that these two are linearly independent because there is no scalar that you can multiply one by to get the other. That is, for $\alpha \in \mathbb{R}$, the equation $$\alpha\begin{pmatrix} 1\\ 0 \end{pmatrix} = \begin{pmatrix} 0\\ 1 \end{pmatrix}$$ has no solutions. This is a good rule for any set of two vectors, and is in fact equivalent to the formal definition for if there were scalars $\alpha_{1}, \alpha_{2} \in \mathbb{R}$ such that $$\alpha_{1}\begin{pmatrix} 1\\ 0 \end{pmatrix} + \alpha_{2}\begin{pmatrix} 0\\ 1 \end{pmatrix} = 0$$ (which is just the formal definition), then we could rearrange it to get $$-\frac{\alpha_{1}}{\alpha_{2}}\begin{pmatrix} 1\\ 0 \end{pmatrix} = \begin{pmatrix} 0\\ 1 \end{pmatrix}.$$ This is clearly false!

The formal definition for an arbitrary number of vectors is an extension of this idea. We pick $0$ to be the right hand side for convenience, since every vector space must have the zero vector (by definition), so that if some vectors affect each other in some way, they must be able to be combined some way to give you the zero vector. In fact, if a set of vectors are linearly independent, then the only vector that they agree on is the zero vector, and it can only be obtained by multiplying each vector by zero (and summing them up).

If they can be combined in some way to give you the zero vector, then you can just rearrange them (like above) and find one of the vectors as a linear combination of the others, which would clearly imply that it is not independent of the others.

To make this more clear, let $\{v_{1}, v_{2}, \ldots, v_{n}\}$ be a set of vectors, and suppose there exists non-zero scalars $\alpha_{i} \in \mathbb{R}$ such that $$\alpha_{1}v_{1} + \alpha_{2}v_{2} + \ldots + \alpha_{n}v_{n} = 0.$$ Then you can just rearrange this to get, for example, the equation $$\frac{-\alpha_{1}}{\alpha_{n}}v_{1} + \frac{-\alpha_{2}}{\alpha_{n}}v_{2} + \ldots + \frac{-\alpha_{n-1}}{\alpha_{n}}v_{n-1} = v_{n}.$$ Then $v_{n}$ depends on the other vectors, so the set is not linearly independent! Thus, the only way that you can get a set of linearly independent vectors to all give you the zero vector, is that all must be multiplied by zero, else you can rearrange the equation to show that one of the vectors depends on the others, which would be a contradiction. This is why the definition is the way it is.

As a note for your third point, the definition wants $\alpha_{i} = 0$ for all $i = 1, \ldots, n$. That is, every coefficient has to be zero, not just one of them.

Your intuition may be leading you astray. You write "linear independence is where the variables are independent and don't affect each other." But vectors are not variables. $(1, 0)$, $(0, 1)$, and $(1, 1)$ are linearly dependent, but that's because you can get $(1, 1)$ by adding the first two vectors together. It's not because changing one of the vectors will change the other, since they are what they are.

The idea of one vector being dependent on others is that you can get the first vector by scaling the other vectors appropriately and then adding the scaled vectors together. For instance, $(2, 4)$ is dependent on $(1, 0)$ and $(0, 1)$ because $$2 \cdot (1, 0) + 4 \cdot (0, 1) = (2, 4)$$ Since you can represent $(2, 4)$ by multiplying $(1,0)$ and $(0,1)$ by constants and adding the results together, it is not independent of them.

But you could take the equation and move the $(2,4)$ to the other side, like this: $$2 \cdot (1, 0) + 4 \cdot (0, 1) + (-1) \cdot(2,4) = 0$$ And that's an example of just what the definition is saying. When you say that you can get a sum like that to add up to zero, it's the same as saying that you can get any one of the vectors by multiplying the others by appropriate constants and adding them. And that's what it means to not be independent.

Each of $n$ vectors in the collection determines a line. If it is possible to find $n$ men of various strength that, when they pull or push the point along those lines, the point does not move, then the vectors are dependent. It is because one of those men can neutralize the collective effort of all others. In other words, it is possible to apply nonzero forces along the lines to balance the system.

If the only possibility for the point to stay still is to apply zero forces along the lines then the vectors are independent, i.e. each direction cannot be affected/neutralized by others.

• Some of those men look kind of young and/or female. ;) May 14, 2018 at 4:06
• @Obie2.0 Yes, it is a total gender and age equality in the vector world.
– A.Γ.
May 14, 2018 at 4:26

I hesitate to add yet another answer, but nowhere do I see the geometric interpretation clearly mentioned.

You know that two points determine a line and three points determine a plane. If higher dimensional geometry is allowed, this pattern continues: 4 points determine a 3-plane (3 dimensional hyperplane = a space), 5 points determine a 4-plane, etc.

But, of course, that is a synopsis that passes over a very important qualification: 3 points determine a plane only if they are not co-linear. 4 points determine a 3-plane only if they are not coplanar. So to be explicit, we define the concept of "general position": A collection of $n+1$ points is in general position if none of the points lies in the smallest linearly-closed set containing the remaining points ("linearly-closed" meaning that given any two distinct points in the set, the entire line through the points is also in the set). $n+1$ points determine a unique $n$-plane if and only if they are in general position. Otherwise they only determine a $k$-plane for some $k < n$.

This is exactly what is going on with linear dependence/independence. With vector spaces, we have one point given to us gratis: the origin. The vectors in the set whose dependence we are examining are the remaining geometric points. If a set of two vectors is linearly dependent, then one of the vectors lies on the line determined by the other vector and the origin (and vice versa), so the three points (origin and two vectors) are not in general position. Three vectors are linearly dependent if at least one of them lies in the plane (i.e., the span, which is the smallest vector subspace) determined by the other two and the origin, and are linearly independent if each vector does not lie in the span of the other two. Et cetera.

For the intuition of the general case let consider the simplest case $V=\mathbb{R^2}$ and the two sets

• $S=\{(1,0),(1,1),(0,1)\}$

• $T=\{(1,0),(1,1)\}$

and apply the definitions, that is for $S$

• $1\cdot (1,0)-1\cdot (1,1)+1\cdot (0,1)=(1-1,-1+1)=(0,0)$

and for $T$

• $c_1(1,0)+c_2(1,1)=(c_1+c_2,c_2)=(0,0) \implies c_1=c_2=0$
• "Applying the definitions" is the opposite of what is meant by intuition May 13, 2018 at 21:09
• @dbx Recall that you can give your own answer if you think there are best methods to explain.
– user
May 13, 2018 at 21:14
• @dbx Apply the definition to a simple case can be very useful for the intuition of the general case.
– user
May 13, 2018 at 21:15
• @dbx Note also that the OP is asking for "Intuition for Formal Definition" and not "Intuition for Linear Independence/Dependence"
– user
May 13, 2018 at 21:19
• @LancePollard Let me know if my answer can be useful to you, otherwise I can delete that. Thanks
– user
May 13, 2018 at 21:20