I am having troubles understanding what is going on here. Would anyone be able to do this step by step with values so that I will be able to understand the SUM and how it works.

xi is in this case colum 1, and Matrix X is the entire thing.

Matrix XMatrix X

Also note that looking at the large general matrix you it should have a diagonal with sum of squares and off-diagonal is the cross product.

enter image description here

Also to clarify X`X(where 'is the transpose) is equal to the one above. This I am not able to understand.

To Clarify the question, It is stated in the book that X'X should be equal to the sumof xixi'. --> as shown in the picture. however if I take Matrix(as given above) X'X i get {{4,14},{14,54}}. When I try this with the sum function enter image description here I am not able to get the same result, could someone please be so kind to take the effort writing this example out step by step.

  • $\begingroup$ AUGH that is notation is terrible! I'm sorry you have to slog through that mess of summations. $\endgroup$ – Eric Stucky Dec 29 '13 at 5:33
  • $\begingroup$ The book skips a very important step. It should read: $\sum_1^N (\text{product of column and row vector}) = \sum_1^N (\text{matrix of products}) = \text{matrix of sums of products}$. The missing step is that the sum distributes inside the matrix (this is just because of how the addition of matrices is defined—you just add the individual components). $\endgroup$ – Slade Dec 29 '13 at 6:45
  • $\begingroup$ Could you please show how this is done with an example? $\endgroup$ – ALEXANDER Dec 29 '13 at 7:17

The setup described below seems to be purely formal. But it is motivated by the underlying conceptual content, as explained in linear algebra courses.

A matrix $A=[a_{ik}]$ is with $m$ rows and $n$ columns is called an $(m\times n)$-matrix. When an $(m\times n)$ matrix $A$ and an $(n\times p)$-matrix $B$ are given (note the $n$ appearing twice here) then their product $AB$, an $(n\times p)$ matrix, is defined by $${\rm elm}_{jl}(AB):={\rm row}_j(A)\cdot {\rm col}_l(B):=\sum_{k=1}^n a_{jk}\>b_{kl}\qquad(1\leq j\leq m, \>1\leq l\leq p)\ .\tag{1}$$ Here the $\cdot$ denotes the usual scalar product between tuples of the same length.

An $n$-tuple ${\bf x}=(x_1,x_2,\ldots, x_n)\in{\mathbb R}^n$ is by convention considered as a column vector, i.e., an $(n\times 1)$-matrix, with the $x_k$ as entries, and one writes ${\bf x}'$ for the analogous row vector, i.e., the $(1\times n)$-matrix with the $x_k$ in its single row.

Given two vectors ${\bf x}$ and ${\bf y}$ both products ${\bf x}'{\bf y}$ and ${\bf x}{\bf y}'$ are defined since the "compatibility condition" is fulfilled in both cases. The result of the first multiplication is a $(1\times 1)$-matrix, which is the same as a number. This number is nothing else but the usual scalar product of the two vectors ${\bf x}, {\bf y}\in{\mathbb R}^n$.

The result of the second multiplication however is an $(n\times n)$-matrix whose elements according to $(1)$ are given by $${\rm elm}_{jl}({\bf x}{\bf y}')=\sum_{k=1}^1 {\bf x}_{jk}\>{\bf y}'_{kl}=x_j\>y_l\qquad(1\leq j\leq n, \>1\leq l\leq n)\ .\tag{2}$$ In your case you are given a list of $N$ vectors $${\bf x}_i=(x_{i.1},x_{i.2},\ldots, x_{i.K})\in{\mathbb R}^K\qquad(1\leq i\leq N)\ .$$ The index $i$ does not number any components, but denotes the position of the vector ${\bf x}_i$ in the list, and I refer to the components after the dot.

According to the above each product of the form ${\bf x}_i{\bf x}'_i$ $\>(1\leq i\leq N)\ $ is a $(K\times K)$-matrix whose elements according to $(2)$ are given by $${\rm elm}_{jl}({\bf x}_i{\bf x}_i')=x_{i.j}\>x_{i.l}\qquad(1\leq j\leq K, \>1\leq l\leq K)\ .\tag{2}$$ The sum $S:=\sum_{i=1}^N{\bf x}_i{\bf x}_i'$ of these $N$ matrices is to be taken elementwise. Therefore it is again a $(K\times K)$-matrix, whose elements are given by $${\rm elm}_{jl}(S)=\sum_{i=1}^N x_{i.j}\>x_{i.l}\ ,$$ as displayed in your question.

Consider the following numerical example: We are given the two vectors (number triples) $${\bf x}_1:=(2,3,5),\quad {\bf x}_2:=(1,1,1)\ .$$ This corresponds to $N=2$ and $K=3$. Then $${\bf x}_1{\bf x}'_1=\left[\matrix{2\cr 3\cr 5\cr}\right]\>[2\ 3\ 5]= \left[\matrix{4&6&10\cr 6&9&15\cr 10&15&25\cr}\right]\ ,\qquad {\bf x}_2{\bf x}'_2=\left[\matrix{1\cr 1\cr 1\cr}\right]\>[1\ 1\ 1]= \left[\matrix{1&1&1\cr 1&1&1\cr 1&1&1\cr}\right]\ .$$ It follows that $$S=\sum_{i=1}^2 {\bf x}_i{\bf x}'_i=\left[\matrix{5&7&11\cr 7&10&16\cr 11&16&26\cr}\right]\ .$$ The "$10$" in the center of the matrix $S$ results from $9+1=x_{1.2}^2+x_{2.2}^2$, as indicated in the displayed source.

| cite | improve this answer | |
  • $\begingroup$ Please would you be able to show this by the numbers that I have posted? $\endgroup$ – ALEXANDER Dec 30 '13 at 12:10
  • $\begingroup$ How do I go about computing the sum of these N matrices? You see that X´X gives me a 2 by 2 matrix, while the sum xi.xi´gives me a 3 by 3 matrix. Where I distribute the sum inside the matrix and where the values of the diagonal is squared. I am not however able to see how the sum works to get me to a 2 by 2 matrix. Hope you are able to help. Regards Aleksander $\endgroup$ – ALEXANDER Dec 30 '13 at 12:35
  • $\begingroup$ @ALEXANDER: See my edit. I don't see $2$ and $3$ in the displayed source, but $K$ and $N$, as in my text. $\endgroup$ – Christian Blatter Dec 30 '13 at 15:23
  • $\begingroup$ Sorry I thought that I got it but no,, I see that if the sum goes from 1 to 3 then xijxil would be 2*2+3*3+4*4+5*5 = would equal to 54. But I am not able to see how I get 4 and 14 In {{54,14},{14,4}}. I am almost giving up now. Could you please go by the numbers? $\endgroup$ – ALEXANDER Dec 31 '13 at 7:51
  • $\begingroup$ @ALEXANDER: See my appendix. This is my last word in this matter. $\endgroup$ – Christian Blatter Dec 31 '13 at 9:20

We know that a column vector multiplied by a row vector is a matrix (not to be confused with the row vector by a column vector, which is the inner product). The transpose of a vector essentially 'converts' it into another alignment of vector. Thus:

$$\vec X = \left( \begin{array}{ccc} a \\ b \\ c \\ \end{array} \right) \\ \vec X' = \left( \begin{array}{ccc} a & b & c\end{array} \right)$$

Reviewing sigma notation, we're adding up all the terms $n$ times, and $i$ counts it. It repeats each time until the $i$ is equal to the initial $i_0 + n$. Thus, for $X\cdot X'$, we multiply the two vectors and distribute the sigma to each component.

| cite | improve this answer | |
  • $\begingroup$ I do not get it, in the book it states that X´X is equal to sum[xi*xi`] $\endgroup$ – ALEXANDER Dec 29 '13 at 5:56
  • $\begingroup$ Would you be able to show it step by step with the numbers I have given. When I calculate X`X I get {{54,14},{14,4}} however when trying to do it with sum I get it all wrong. $\endgroup$ – ALEXANDER Dec 29 '13 at 5:58

Given two column vectors a and b, the Euclidean inner product and outer product are the simplest special cases of the matrix product, by transposing the column vectors into row vectors.

The inner product is the normal product which will yield a scalar.

The outer/ tensor product however,is a row vector multiplied on the left by a column vector: see the link below :-


The multiplication given in the question is an outer/ tensor product.

The vector 'xi' can be interpreted as a rank 1 tensor with dimension K, and the vector 'xiT' as a rank 1 tensor with dimension K. The result is a rank 2 tensor with dimension (K, K).

And Sigma sign gets distributed inside the (K,K) matrix.

| cite | improve this answer | |
  • $\begingroup$ Would you be able to show it step by step with the numbers I have given. When I calculate X`X I get {{54,14},{14,4}} however when trying to do it with sum I get it all wrong. $\endgroup$ – ALEXANDER Dec 29 '13 at 6:34
  • $\begingroup$ I am having some problem uploading images in this site. However please see this link upload.wikimedia.org/math/9/2/c/… . Note the tensor multiplication of two matrices in the given link. In the same way you can get the multiplication done for Xi matrix . You need to multiply whole of the 'xiTranspose' matrix with each element of xi matrix and distribute the sigma in all the elements thereafter. Note:- see also this link en.wikipedia.org/wiki/Kronecker_product $\endgroup$ – anjan Dec 30 '13 at 8:13
  • $\begingroup$ I have edited the question, I am not able to see what you are stating because when given a example I am not able to replicate it, would you have a look at the edited version and then see if you are able to work out the example, I would appreciate it so much. I have been struggling so hard trying to understand it but Im not able to replicate it. $\endgroup$ – ALEXANDER Dec 30 '13 at 9:37
  • $\begingroup$ Can you post full snapshot of the problem because i think answer must be XX' instead of X'X . This is because XX' is a 4 x 4 matrix same as the xixi' matrix which will also be a 4 x 4 matrix. See the working of the above matrix in this link ->drive.google.com/file/d/0B5FWdKs3BipkTEZ4VmRFSmI4elE/… NOTE :- In your snapshot K=4 as there are only xi is vector of 4 elements. which means xixi' will be a 4x4 matrix. $\endgroup$ – anjan Dec 31 '13 at 3:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.