19
$\begingroup$

I seemed to have thoroughly confused myself today...

Long story short, the question is simple. Is matrix multiplication just a special case of the dot product of two sets of vectors when the sets of vectors have the same cardinality and all vectors in both sets have the same length?

I assume the answer is yes from reviewing the computation of matrix multiplication and the dot product.

$\endgroup$

5 Answers 5

24
$\begingroup$

Dot product is defined between two vectors.

Matrix product is defined between two matrices.

They are different operations between different objects.

The connection between the two operations that comes to my mind is the following: To calculate the $c_{i,j}$ entry of the matrix $C:=AB$, one takes the dot product of the $i$'th row of the matrix $A$ with the $j$'th column of the matrix $B.$

$\endgroup$
2
  • 3
    $\begingroup$ Aren't vectors a special case of matrices? $\endgroup$
    – q0mlm
    Oct 26, 2020 at 11:33
  • $\begingroup$ @mlm0b11011 They are. $\endgroup$
    – Pawel
    Oct 27, 2020 at 18:28
6
$\begingroup$

$$ A^TB \ \equiv A \bullet B \iff A \ \text{and} \ B \ \text{are} \ n \times 1 \ \text{matrices}. $$

So you could think of a dot product as a special case of matrix multiplication.

$\endgroup$
5
$\begingroup$

What @Pawel said, additionally, though, I would like to add that there is a nice duality between $1\times 2$ matrices and 2d vectors.

3Blue1Brown covers this in the 9th episode of his series 'Essence of Linear Algebra'

$\endgroup$
2
  • 1
    $\begingroup$ Back around the time I original asked this question I found and began watching 3Blue1Brown religiously. His Linear Algebra videos are amazing. $\endgroup$
    – KDecker
    Jan 30, 2021 at 22:00
  • $\begingroup$ @KDecker I couldn't agree more. $\endgroup$ Jan 30, 2021 at 22:12
4
$\begingroup$

There is a difference between vectors and matrices and also a connection

Vector

A vector is an object that has both a magnitude and a direction. Example Force and Velocity. Both have magnitude as well as direction. However, we need to specify also a context where this vector lives -Vector Space. For example, when we are thinking about something like a Force vector, the context is usually 2D or 3D Euclidean world.

The easiest way to understand the Vector is in a geometric context, say 2D or 3D cartesian coordinates, and then extrapolate it for other Vector spaces which we encounter but cannot really imagine.

Vectors are represented as matrices. But a matrix is just a rectangular array of numbers and nothing else.

This matrix below is an example of a Euclidean Vector in three-dimensional Euclidean space (or $R^3$). So a vector is represented as a column matrix or a row matrix.

$$ a = \begin{bmatrix} a_{1}\\a_{2}\\a_{3}\ \end{bmatrix} = \begin{bmatrix} a_{1} & a_{2} &a_{3}\end{bmatrix} $$

Dot product has a specific meaning. Matrix multiplication has no specific meaning, than may be a mathematical way to solve system of linear equations Why, historically, do we multiply matrices as we do?

Coming back to dot product - Algebraically, the dot product is the sum of the products of the corresponding entries of the two sequences of numbers.

$ \text{If } \vec a = \left\langle {a_1,a_2,a_3} \right\rangle \; \text{and } \vec b = \left\langle {b_1,b_2,b_3} \right\rangle \; \text{then } \; \vec a \cdot \vec b = {a_1}{b_1} + {a_2}{b_2} + {a_3}{b_3}$

Which if we write in matrix form, we need to mathematically take the transpose of a vector and do 'matrix' multiplication to get the above dot product.

So coming back full circle to the question - matrix multiplication is a tool to find vector dot product (assuming we are talking about matrices in the context of vectors)

$$ a = \begin{bmatrix} a_{1}\\a_{2}\\a_{3}\ \end{bmatrix} b = \begin{bmatrix} b_{1}\\b_{2}\\b_{3}\ \end{bmatrix} a.b = ab^T=\begin{bmatrix} a_{1}b_{1} & a_{2}b_{2} &a_{3}b_{3}\end{bmatrix} $$

https://mathinsight.org/dot_product_matrix_notation

Note

Geometrically dot product of two Euclidean vectors are the Euclidean magnitudes of the two vectors and the cosine of the angle between them

$\vec a \cdot \vec b = \left\| {\vec a} \right\|\,\,\left\| {\vec b} \right\|\cos \theta$

This is simple to visualize in Euclidean space but is true for all dimensions. If two vectors are in the same direction the dot product is positive and if they are in the opposite direction the dot product is negative. This can be visualized geometrically putting in the value of the Cosine angle.

So we could use the dot product as a way to find out if two vectors are aligned or not; which tends itself to more interesting uses

$\endgroup$
2
  • $\begingroup$ I beg to differ with your statements of "a matrix is just a rectangular array of numbers " and "matrix multiplication has no specific meaning". This is just as true for vectors as well: "they're just a bunch of numbers and in that context the dot product has no specific meaning." Similarly, matrices also have a geometric interpretation. A matrix represents a linear map / transformation and in that context matrix multiplication has the meaning of composing those maps right to left. $\endgroup$ Sep 18, 2023 at 8:34
  • 1
    $\begingroup$ A matrix can represent different concepts. As a means to encode linear transformation or geometric transformation; or maybe some other concepts. In the context of this answer, what I meant was to encode a set of vectors and to find dot product easily via matrix multiplication (numpy.matmul used to implement nerualnets) $\endgroup$ Sep 19, 2023 at 2:24
0
$\begingroup$

<Matrix multiplication(product)>

  • Matrix multiplication is the multiplication of 2D or more D tensors(arrays). *Either of 2 operands can be a 1D array(tensor) but not both of them.
  • The rule which you must follow to do matrix multiplication is the number of the columns of A tensor(array) must match the number of the rows of B tensor(array).

2D arrays:

    <A>            <B>
[[a, b, c], x [[g, h, i, j], = [[ag+bk+co, ah+bl+cp, ai+bm+cq, aj+bn+cr],
 [d, e, f]]    [k, l, m, n],    [dg+ek+fo, dh+el+fp, di+em+fq, dj+en+fr]]
               [o, p, q, r]]
2 rows         (3) rows
(3) columns    4 columns

[[2, 7, 4], x [[5, 0, 8, 6], = [[35, 58, 59, 69],
 [6, 3, 5]]    [3, 6, 1, 7],    [44, 38, 96, 67]]              
               [1, 4, 9, 2]]   [[2x5+7x3+4x1, 2x0+7x6+4x4, 2x8+7x1+4x9, 2x6+7x7+4x2]
                                [6x5+3x3+5x1, 6x0+3x6+5x4, 6x8+3x1+5x9, 6x6+3x7+5x2]]

In PyTorch with @, matmul() or mm():

import torch

tensor1 = torch.tensor([[2, 7, 4], [6, 3, 5]])
tensor2 = torch.tensor([[5, 0, 8, 6], [3, 6, 1, 7], [1, 4, 9, 2]])

tensor1 @ tensor2  # tensor([[35, 58, 59, 69], [44, 38, 96, 67]])
torch.matmul(tensor1, tensor2) # tensor([[35, 58, 59, 69], [44, 38, 96, 67]])
torch.mm(tensor1, tensor2) # tensor([[35, 58, 59, 69], [44, 38, 96, 67]])

*Memos:

  • @ or matmul() can multiply 1D or more D tensors(arrays) by dot or matrix multiplication.
  • mm() can multiply 2D tensors(arrays) by matrix multiplication.

A 1D and 3D array. *B 3D tensor(array) has 3 2D tensors(arrays) which have 2 rows and 4 columns each:

  <A>             <B>
[a, b]   x   [[[c, d, e, f], = [[(ac+bg), (ad+bh), (ae+bi), (af+bj)],
               [g, h, i, j]],   [(ak+bo), (al+bp), (am+bq), (an+br)],
              [[k, l, m, n],    [(as+bw), (at+bx), (au+by), (av+bz)]]
               [o, p, q, r]],
              [[s, t, u, v],
               [w, x, y, z]]]
1 row         (2) rows
(2) columns   4 columns

[2, 7]   x   [[[6, 3, 5, 2], = [[47,  6, 66, 32],
               [5, 0, 8, 4]],   [20, 68, 65, 35],
              [[3, 6, 1, 0],    [42, 39, 21, 69]]
               [2, 8, 9, 5]],  [[2x6+7x5, 2x3+7x0, 2x5+7x8, 2x2+7x4],
              [[7, 2, 0, 3],    [2x3+7x2, 2x6+7x8, 2x1+7x9, 2x0+7x5],
               [4, 5, 3, 9]]]   [2x7+7x4, 2x2+7x5, 2x0+7x3, 2x3+7x9]]

In PyTorch with @ or matmul():

import torch

tensor1 = torch.tensor([2, 7])
tensor2 = torch.tensor([[[6, 3, 5, 2], [5, 0, 8, 4]],
                        [[3, 6, 1, 0], [2, 8, 9, 5]],
                        [[7, 2, 0, 3], [4, 5, 3, 9]]])

tensor1 @ tensor2
# tensor([[47, 6, 66, 32], [20, 68, 65, 35], [42, 39, 21, 69]])
torch.matmul(tensor1, tensor2)
# tensor([[47, 6, 66, 32], [20, 68, 65, 35], [42, 39, 21, 69]])

<Dot multiplication(product)>

  • Dot multiplication is the multiplication of 1D tensors(arrays).
  • The rule which you must follow to do dot multiplication is the number of the rows of A and B tensor(array) must be 1 and the number of the columns must be the same.
   <A>         <B>
[a, b, c] x [d, e, f] = ad+be+cf
1 row       1 row
3 columns   3 columns

[2, 7, 4] x [6, 3, 5] = 53
                 (2x6)+(7x3)+(4x5)
  [2, 7, 4]
   x  x  x
  [6, 3, 5]
      ||
 [12, 21, 20]
 12 + 21 + 20
      ||
      53

In PyTorch with @, dot() or matmul(). *dot() can multiply 1D tensors(arrays) by dot multiplication:

import torch

tensor1 = torch.tensor([2, 7, 4])
tensor2 = torch.tensor([6, 3, 5])

tensor1 @ tensor2 # tensor(53)
torch.dot(tensor1, tensor2) # tensor(53)
torch.matmul(tensor1, tensor2) # tensor(53)
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .