# Significance of Matrix-Vector multiplication

Can someone give me an example illustrating physical significance of the matrix-vector multiplication?

1. Does multiplying a vector by matrix transforms it in some way?
2. Do left & right multiplication signify two different things?
3. Is matrix a scalar thing? (EDIT)

Thank you.

-

The physical significance depends on the matrix. The primary point is that multiplication by a matrix (in the usual sense, matrix on the left) represents the action of a linear transformation. We'll work with one basis throughout which we'll use to represent our matrices and our vectors. Just note that because of linearity, if we have a vector $x = c_1e_1 + \ldots + c_ne_n$, and a linear transformation $L$, then $$\begin{eqnarray*} L(x) &=& L(c_1e_1 + \ldots + c_ne_n) \\ &=& L(c_1e_1) + \ldots + L(c_ne_n) \\ &=& c_1L(e_1) + \ldots + c_nL(e_n) \end{eqnarray*}.$$

This means that any linear transformation is uniquely determined by its effect on a basis. So to define one, we only need to define its effect on a basis. This is the matrix

$$\left(L(e_1) \ldots L(e_n)\right) = \left( \begin{array}{c} a_{11} & \ldots & a_{1n} \\ \ldots & \ldots & \ldots \\ a_{n1} & \ldots & a_{nn} \end{array} \right)$$

where $a_{ij}$ is the $i$'th compenent of $L(e_j)$.

Let's call this matrix $M_L$. We want to define multiplication of $M_L$ and some vector $x$ so that $M_L \cdot x = L(x)$. But there's only one way to do this. Because the $j$'th column of $M_L$ is just $L(e_j)$ and in light of our decomposition of the action of $L$ in terms of the $L(e_j)$, we can see that

$$M_L \cdot x = \left( \begin{array}{c} a_{11} & \ldots & a_{1n} \\ \ldots & \ldots & \ldots \\ a_{n1} & \ldots & a_{nn} \end{array} \right) \cdot \left( \begin{array}{c} c_1 \\ \ldots \\ c_n \end{array} \right)$$

must equal

$$c_1\left( \begin{array}{c} a_{11} \\ \ldots \\ a_{n1} \end{array} \right) + \ldots + c_n\left( \begin{array}{c} a_{1n} \\ \ldots \\ a_{nn} \end{array} \right) = \left( \begin{array}{c} c_1a_{11} + \ldots + c_na_{1n} \\ \ldots \\ c_1a_{n1} + \ldots + c_na_{nn} \end{array} \right)$$

which is the standard definition for a vector left-multiplied by a matrix.

EDIT: In response to the question "Is a matrix a scalar thing". Kind of but no.

If you consider the most basic linear equation in one variable, $y = mx$, where everything in sight is a scalar, then a matrix generalizes the role played by $m$ to higher dimensions and a vector generalizes the role played by $y$ and $x$ to higher dimensions. But matrices don't commute multiplicatively. So that's one big thing that's different. But they're strikingly similar in a lot of ways. We can define the function of matrices $f(A) = A^2$ and we can differentiate it with respect to $A$. When we do this in one variable with the map $f(x) = x^2$, we get the linear map $f_x'(h) = 2xh$ but when we do it with matrices, we get the linear map $f_A'(H) = AH + HA$. If matrices commuted, then that would just be $2AH$!

EDIT2:

My "add comment" button isn't working for some reason. The $e_j$'s are a basis, $e_1, \ldots, e_n$. I think the best thing to do would be to wait for your teacher to get around to it. I sometimes forget that people introduce matrices before vector spaces and linear transformations. It will all make much more sense then. The main point of a basis though is that it's a set of vectors so that every vector in the given space can be written as a unique linear combination of them.

-
what do you mean by e's in the definition of vector of x? (Excuse me, I've an idea of vectors which're matrices of (nx1) or (1xn) dimension, So, I'm finding it difficult to get this representation of vectors) AND what do you mean by 'basis' (really I'm poor on this, so). –  Amit L Apr 4 '11 at 9:11
hey, unfortunately, I've no teacher for this & I'm trying to do this on my own (the linear algebra, probability & statistics for machine learning, which too I'll try to self-learn). So would you suggest some material that will satisfy my need, provided I'm in the situation mentioned? –  Amit L Apr 4 '11 at 9:38
@Amit I would highly recommend amazon.com/… I can't think of a better book on linear algebra. –  knucklebumpler Apr 4 '11 at 9:45
@Amit, I also just found a pdf which I'll link to. I can't vouch for it but from looking over it, it looks excellent: math.ucdavis.edu/~anne/mat67_course_notes.pdf –  knucklebumpler Apr 4 '11 at 9:50
thanks. How about that Gilbert Strang Book? MIT prof., I guess...? –  Amit L Apr 4 '11 at 9:59

I will interpret physical significance geometrically.

1. Does multiplying a vector by matrix transforms it in some way?

Yes. It performs a sequence of translations, rotations, and scalings. If you keep studying linear algebra you will learn how to write down the formulas for each of these operations given a matrix.

1. Do left & right multiplication signify two different things?

Yes: only one of these is defined. If you have an $m\times n$ matrix $M$ and a vector $V\in\mathbb{R}^n$ then only the multiplication $M V$ is defined.

1. Is matrix a scalar thing? (EDIT)

A matrix is most definitely not a scalar.

-
then is matrix a vector of vectors, possibly? Like 3x2 matrix, a row vector of two column vectors (each of length 3) or a column vector of three row vectors (each of length 2) ? Help! –  Amit L Apr 4 '11 at 9:03
Yes, you can think about it like this. There are a lot of equivalent ways to think about it :). –  Glen Wheeler Apr 4 '11 at 12:47

I believe that parts 2 and 3 of your question have been answered well. I'd like to take a stab at part 1, though the other answers to this part are probably better.

There's an interesting way of thinking of the application of a matrix to a vector using the Singular Value Decomposition (SVD) of a matrix. Let A be an $m \times n$ rectangular matrix. Then, the SVD of A is given by $A = U \Sigma V^T$, where $U$ is an $m \times m$ unitary matrix, $\Sigma$ is an $m \times n$ diagonal matrix of so-called singular values and $V$ is an $n \times n$ unitary matrix. For more on the SVD, check out the Wikipedia article: http://en.wikipedia.org/wiki/Singular_value_decomposition

That same article contains proof that every matrix has an SVD. Given that fact, we can now think of matrix vector multiplication in terms of the SVD. Let $\bf x$ be a vector of length $n$. We can write the matrix-vector multiplication as ${\bf b} = A {\bf x}$. But, $A{\bf x} = U\Sigma V^T {\bf x}$.

Since $V$ is unitary, $V {\bf x}$ does not change the magnitude of ${\bf x}$. Unitary matrices applied to a vector only change the direction of the vector (rotate it by some angle). The product $V^T {\bf x}$ rotates ${\bf x}$.

$\Sigma$ is a diagonal matrix. Its entries directly multiply the corresponding entries of the vector ${\bf x}$, thus scaling the vector (increasing its length) along the axis around which $V$ rotated ${\bf x}$. However, remember that the rotated vector $V^T{\bf x}$ is a vector of length $n$ while $\Sigma$ has dimensions $m \times n$. This means that $\Sigma$ is also embedding the vector $V^T{\bf x}$ in an $m$-dimensional space, i.e., changing the dimensions of the vector. If $m=3$ and $n=2$, for example, $\Sigma$ scales the 2D vector $V^T{\bf x}$ in 2 dimensions and then "places" it in a 3D space.

Finally, we have the product of the unitary matrix $U$ with the $m$-dimensional vector $\Sigma V^T{\bf x}$. $U$ rotates that vector in the $m$-dimensional space.

Every matrix thus potentially rotates, scales and embeds and then again rotates a vector when applied to a vector. When $m=n$, of course, a matrix-vector product doesn't involve any embedding- simply a rotation, scaling and another rotation. Like a little assembly line.

-