Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Suppose a centered configuration $X\in\mathbb{R}^{n\times 2}$ is rotated to two principal components, i.e, $X^TX=diag(\lambda_1, \lambda_2)$, where the diagonal entries are the variance across the $x$ and $y$ axes, respectively.

Now, suppose that the configuration is rotated, i.e, $X'=XR$, for $R\in\mathbb{R}^{2\times 2}$. To rotate $X'$ back to principal components, one performs eigendecomposition on $$K=(X')^TX'=R^T(X^TX)R=R^T\cdot diag(\lambda_1, \lambda_2)\cdot R,$$ and performs $X'U_K$, where $U_k$ contains eigenvectors of $K$ as its columns (ordered according to non-increasing eigenvalues of $K$);(please correct me if I'm wrong in this PCA recipe).

The above PCA recipe would mean that, when $X'$ is already rotated to principal components, $K=I$, and nothing would change. This raises the following question: what if the variance across $x$ and $y$ are the same, i.e, $X^TX=c\cdot I$? Then, any arbitrary rotation on such $X$, i.e, $X'=XR$ would yield $K=c\cdot I$, and $U_K=I$, hence nothing would be changed. This means that any rotation of such $X$ is a rotation to principal components (hence, infinitely many!).

Furthermore, the first principal component would be the first column of each such $X'$ (which is arbitrary rotated $X$, with the property $X^TX=c\cdot I$).

Is the above reasoning fine?

share|improve this question
    
is $X$ centered row- or columnwise? If rowwise, then the columns were equal except the sign... –  Gottfried Helms Nov 5 '12 at 10:24
    
@GottfriedHelms Assumption behind PCA is that a configuration is column centered: entries of each column sum to zero. –  user506901 Nov 5 '12 at 10:31
add comment

1 Answer 1

up vote 2 down vote accepted

The rotation-criterion for principal components is for your example $$ \sum x°^2 = max ; \sum y°^2 =min $$ where $x$ denotes the first and $y$ the second column and the superscript-circle that we consider the values when rotated by the sought rotation-angle. I also omit the row-indices.
Because the squaring leads to nonnegative values only we can also rewrite the criterion $$ max = \sum x°^2-y°^2 $$ (After that we could also check that the orthogonality criterion shall also be satisfied $ 0 = \sum 2 x° y° \quad $ ).


If the cosine and sine of the rotation-angle are denoted with c and s then $x° = xc-ys \quad $ and $y°=yc+xs \quad $ and we can rewrite the PC-criterion as $$ \begin{eqnarray} max &=& \sum (xc-ys)^2-\sum (yc+xs)^2 \\ &=& \sum (x^2c^2-2 xycs + y^2s^2) - \sum (y^2c^2+2xycs+x^2s^2) \\ &=& (c^2-s^2)\sum (x^2-y^2) - 2cs \sum (2xy) \\ &=& (c^2-s^2)(\sum x^2-\sum y^2) - 2cs \sum (2xy) \\ \end{eqnarray} $$ But if now the eigenvalues are equal then the sum-of-squares of the x-coordinates equal already that of the y-coordinates and we have $$ \begin{eqnarray} max &=& (c^2-s^2)(0) - 2cs \sum (2xy) \\ \end{eqnarray} $$ and if also the columns are already orthogonal then we have $$ \begin{eqnarray} max &=& (c^2-s^2)(0) - 2cs (0) \\ \end{eqnarray} $$ and c and s can be arbitrary (and thus the rotation-angle) since the rhs is always zero and no rotation can increase that value to a greater max-value.

[update]:
Conclusion : if the two columns are already in principal components position, then $0 =\sum x y$ (orthogonality) and $\sum x^2 = max$ and $\sum y^2=min$ where max and min mean here the maximal and minimal possible values which can be achieved by rotations only.

If now in a given case the eigenvalues ($ \lambda_x= \sqrt{\sum x^2} =\lambda_y= \sqrt{\sum y^2} $ are also equal then a rotation does not change the orthogonality nor the ratio of the eigenvalues, so the result of any rotation can then be called "pincipal components solution" with the same right. (Perhaps we should call such a situatin a "Principal plane" instead)


// Taking ascript in my MatMate-script language for matrix-analysis
// We create a 2-column random matrix, coordinates are centered along the columns
// and sum-of-squares per column is normed to be 1
[12] A=normsp(abwsp(randomu(8,2)))
      A : 
 -0.0147      0.0768
  0.2692     -0.0060
 -0.2468     -0.3572
 -0.0311     -0.4312
  0.0578      0.1702
  0.5288      0.3220
  0.1788     -0.3985
 -0.7419      0.6238

// check correlation and variance by correlation matrix
[13] chkA = A' * A
      chkA : 
  1.0000     -0.2551
 -0.2551      1.0000

// rotate to PC-position     
[14] B = rot(A,"pca")
      B : 
 -0.0647      0.0439
  0.1946      0.1861
  0.0780     -0.4271
  0.2829     -0.3269
 -0.0795      0.1613
  0.1463      0.6016
  0.4082     -0.1554
 -0.9657     -0.0835

// check: the correlations are zero (offdiagonal elements)
// variances are ordered from max/min (diagonal elements)
// thus B is in PC-position
[15] chkB = B' * B
      chkB : 
  1.2551     -0.0000
 -0.0000      0.7449

// now try what happens in a case, where also the ssq (=also squares of eigenvalues)    
// are equal. We just divide columnwise by their "lengthes"      
[16] C = B /# sqrt(sqsumsp(B)#)
      C : 
 -0.0578      0.0508
  0.1737      0.2157
  0.0696     -0.4949
  0.2525     -0.3787
 -0.0710      0.1869
  0.1306      0.6970
  0.3644     -0.1800
 -0.8620     -0.0968

// Ok, orthogonal, columns are uncorrelated ( offdiagonal is 0)
// eigenvalues are equal (diagonal elements)
[17] chkC = C' * C
      chkC : 
  1.0000     -0.0000
 -0.0000      1.0000

// we can rotate by a random-rotation  
// and always get orthogonality and ordered eigenvalues in the diagonal
//   (the entries are equal)    
[18] D = C * gettrans(randomu(2,2),"drei")
      D : 
 -0.0010      0.0769
  0.2764      0.0159
 -0.3203     -0.3836
 -0.1115     -0.4413
  0.0910      0.1780
  0.6045      0.3708
  0.1110     -0.3910
 -0.6501      0.5743

[19] chkD = D' * D
      chkD : 
  1.0000     -0.0000
 -0.0000      1.0000

// another random-rotation:     
[20] D = C * gettrans(randomu(2,2),"drei")
      D : 
 -0.0175      0.0749
  0.2666      0.0747
 -0.2307     -0.4433
 -0.0144     -0.4550
  0.0508      0.1933
  0.5111      0.4916
  0.1921     -0.3582
 -0.7579      0.4218

[21] chkD = D' * D
      chkD : 
  1.0000     -0.0000
 -0.0000      1.0000
share|improve this answer
    
So, you're implying that in such case any rotation is a rotation to principal components, since no greater max value can be achieved. Consequently, the first principal component is the first (or the second) column of any rotated 2D solution? –  user506901 Nov 5 '12 at 12:22
1  
Hmm, consider the group of children in a kindergarten. They all get a sweety, and we define, that that one with the biggest sweety shall be the "principal darling". Now you give all sweeties with the same size. After that, you can "rotate" the sweeties through the hands of the children - who will be the "principal darling"? ;-) –  Gottfried Helms Nov 5 '12 at 12:26
    
Great methaphor and fully contained answer! –  user506901 Nov 5 '12 at 15:51
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.