Questions tagged [principal-component-analysis]

Principal component analysis (PCA) is a linear dimensionality reduction technique. It reduces a multivariate dataset to a smaller set of constructed variables preserving as much information (as much variance) as possible. These variables, called principal components, are linear combinations of the input variables.

Filter by
Sorted by
Tagged with
-1 votes
0 answers
7 views

Multilinear principal component analysis

I was curious to know that when we perform PCA and for example we say that the top 4 principal components capture 95% of the variance. In a similar fashion, how can I do it using MPCA? Is there a ...
Zlatan's user avatar
  • 1
0 votes
0 answers
31 views

Eigenvectors and Eigenvalues of Matrix product with tall matrix with itself

Given tall matrix $B \in \mathbf{R}^{n \times r}$, where $n \gg r$, let $$ A := B B^T $$ and the columns of $B$ are in general not orthogonal to each other. I would like to get the eigenvectors and ...
checkThisOut's user avatar
0 votes
0 answers
30 views

PCA: derivation of successive principal axes

Let $\Sigma$ be the covariance matrix. The first principal axis of PCA, $u_1$, can be found by solving the optimization problem: $$\max_{u_{1}}=u^T_1\Sigma u_1$$ subject to the constraint $u^T_1u_1=1$....
s1m0n3's user avatar
  • 13
1 vote
1 answer
72 views

Does PCA always find the best-fitting plane?

Here, the best-fitting plane is the plane that minimizes the sum of squared perpendicular distance from the data points to the plane. In other words, the best-fitting plane is the solution to the ...
iklnot's user avatar
  • 13
0 votes
0 answers
19 views

Degrees of Freedom in PCA

Suppose we are doing PCA over a historic time series of temperatures. The feature for the PCA to be explained is the time when the temperature was observed every hour. Let’s say for the sake of ...
Identicon's user avatar
1 vote
0 answers
49 views

Confusion regarding the geometrical meaning of singular values in SVD

I am trying to visualize in MATLAB the relationship between the singular value decomposition (SVD) of a matrix of points. To simplify the problem, I am working in 2D and I am considering an ellipse ...
gwn's user avatar
  • 121
0 votes
0 answers
12 views

Principal Component Analysis Based on Scatter Matrices

There is a paper on robust PCA based on scatter matrices: https://wis.kuleuven.be/statdatascience/robust/papers/2005/hubertrousseeuwvandenbranden-robpca-technom-2005.pdf. PCA is generally performed ...
Lyricist's user avatar
1 vote
0 answers
24 views

Canonical Variables and maximization of Covariance [closed]

I've been asked to find the deterministic unitary vectors $a\in \mathbb{R}^p$ and $b \in \mathbb{R}^q$ that maximize $\Cov(a^TX,b^TY)$ for $X$ a $p$-dimensional random vector, and $Y$ a $q$-...
juanan's user avatar
  • 76
0 votes
0 answers
38 views

Proving the relationship between mean-squared error of 1D projection of PCA and the largest eigenvalue of the covariance matrix

I was studying the topic of principal component analysis and came across this problem that I was not able to prove. Consider a data matrix, X and its covariance matrix, S. I know that taking the ...
DataStatsExplorer's user avatar
0 votes
0 answers
14 views

Is this a useful dimensionality reduction on real projective space?

In the following, I will derive a dimensionality reduction for the real projective space. As I will use scalar product as a means of distance, I am unsure if the dimensionality reduction has a useful ...
Niklas Netter's user avatar
0 votes
0 answers
9 views

the first principal component’s variance

The expression comes from The Elements of Statistical Learningp66(3.49) Related part about this is shown in below figure. Where Using SVD, $X$ has the form: $X=UDV^T$. And $D$ is a diagonal matrix, ...
chenqile's user avatar
1 vote
0 answers
161 views

Proof of Eckart-Young Theorem (Mathematics for Machine Learning, Deisenroth)

I am trying to understand a proof of the Eckart-Young Theorem (source in title). Let me add the definition and proof they provided, afterward I'll say what I do not understand. Proof: Some points I ...
Aaron's user avatar
  • 57
1 vote
1 answer
223 views

Why do the "important" eigenvectors of a graph Laplacian have small-magnitude eigenvalues?

In spectral clustering, one computes sample-sample similarities, then from this computes a graph Laplacian matrix. (Typically, one uses the symmetrically normalized Laplacian matrix, but the pattern I'...
calmcc's user avatar
  • 251
0 votes
0 answers
35 views

Should a gram matrix be $R = Q^TQ$ or $R = QQ^T$?

I'm reading a paper about fisher faces, that using kernel PCA together with LDA. The probelm I have is to understand $\Phi(x)$, I know that is an arbitary function for the vector $x$. But I don't ...
euraad's user avatar
  • 2,914
0 votes
0 answers
54 views

How to interpret the diagonalization of a scatter matrix in Principal Component Analysis (PCA)?

Let $X \in \mathbb{R}^{2 \times n}$ be the data matrix, $S_n = (X - \hat{\mu})(X - \hat{\mu})^T$ be the scatter matrix, and $$ S_n = \begin{array}{cc} \begin{bmatrix} 3 & -4 \\ -4 & 3 \end{...
Salman Akbar's user avatar
0 votes
0 answers
54 views

Is there anything to be said on the inverse of the covariance matrix of a reconstructed matrix using PCA or SVD decomposition?

Let $X$ be a real $n \times p$ data matrix where we can assume $n > p$. The SVD decomposition is $X = UDV'$ where $U$ is $n \times p$, $U'U = I_p$, $D$ is $p \times p$ diagonal with singular values ...
Giora Simchoni's user avatar
0 votes
1 answer
111 views

Suppose that the variance-covariance matrix of a $p$-dimensional random vector $X$ is

Suppose that the variance-covariance matrix of a $p$ -dimensional random vector $X$ is $\Sigma=\sigma_{ij}$ for all $i,j=1, 2, ...,p$. Show that the coefficients of the first principal component have ...
RV math's user avatar
  • 67
0 votes
0 answers
19 views

PCA with reverse optimization function

Suppose I want to do a different version of PCA, in which I want to minimize the variances explained. So lets say I find the first such variance minimizing axis $PC^{'}_1$. Then I impose the ...
ztart14578's user avatar
0 votes
0 answers
39 views

Order of the vectors in principal component analysis

For a given matrix $A$, the Principle Component Analysis (PCA) is done by finding the eigenvalues/eigenvectors of the covariance matrix associated with $A$. However, the entries of the covariance ...
user2167741's user avatar
0 votes
0 answers
34 views

PCA on the distance matrix

Let $X$ be a $p$-vector over $n$ individuals and let $X_1, \dots, X_n$ be the $I$−th observation of dimension $p$. Let $D$ be the diagonal matrix of weights $p_i = \frac1n$. Let $W = (w_{i,j} = X_i^T ...
John's user avatar
  • 29
0 votes
0 answers
26 views

PCR as a biased estimator

Ordinary least square estimates for $X_{n \times m}\vec{\beta}=\vec{y}$ $$\vec{\beta}_{OLS}=(X^TX)^{-1}X^T\vec{y}$$ is an unbiased estimator of $\vec{\beta}$ in which it has the smallest variance ...
WeiShan Ng's user avatar
0 votes
0 answers
32 views

Why the number of non-zero singular values is $n-1$ for $X\in \mathbb{R}^{n \times m}$ when $n\ll m$

There's a statement in the paper Phatak (1997) In most spectroscopic problems, the number of non-zero singular values is $n-1$ for $X\in \mathbb{R}^{n \times m}$ when $n\ll m$. Consequently, at most $...
WeiShan Ng's user avatar
0 votes
0 answers
74 views

Proof that PCA is equivalent to MDS when using Euclidean distances

As I was watching a video explaining how MDS works, the narrator mentioned that PCA is equivalent to MDS when Euclidean distances are used. I got confused as to how that's the case. My guess is that ...
baoiba's user avatar
  • 121
1 vote
0 answers
25 views

Relation of principal component analysis between a matrix and its transopose

With a matrix $X_{n\times p}$ ($n>p$), we perform a principal component analysis: $T_{n\times p}=X_{n\times p}W_{p\times p}$ where $W$ is the loadings matrix while $T$ is the scores matrix for $X$. ...
bluepole's user avatar
  • 153
0 votes
0 answers
24 views

Doing PCA with the approach of maximal variance should also give non-orthogonal basis.

Assume we have 3-dimension data. That is set of points (x,y,z) where every x's are very close to either y's or negative of 2y's (i.e., -2y). z is very close to 3 i.e., ranges from 2.9 to 3.1 Example ...
Abc1729's user avatar
  • 53
1 vote
1 answer
78 views

Can SVM be special case of PCA?

Let $X$ and $Y$ two linearly separable finite subsets of a $K$-dimensional real vector space $V$ with orthonormal basis $A = \{a_1,\ldots, a_K\}$. The covariance matrix $\Sigma_A$ of the set $X \cup Y$...
Alberto Carraro's user avatar
0 votes
0 answers
42 views

The coskewness and cokurtosis of uncorrelated standardized random vector

I was conducting the Karhunen-Loeve (K-L) Expansion for a random vector. Based on the KL expansion, I transformed the original random vector into a standardized random vector $\boldsymbol{X}=[X_1,X_2,\...
Alexander's user avatar
1 vote
1 answer
401 views

Why the singular values in SVD are always hierarchical/descending?

Please, I'm trying to understand why singular values (SV) are always hierarchical/descending. At the beginning of my studies, I thought that the hierarchy of sigmas ($ \sigma_1 \geq \sigma_2 \geq ... \...
Caio Rímoli's user avatar
0 votes
1 answer
24 views

Find line with respect to which the moment of inertia is minimized

Consider a function $F(x,y,z): \mathbb{R}^3 \mapsto \mathbb{R}^+$ (i.e., $F(x,y,z) > 0 ~~\forall~ x,y,z$) and consider a set of points in the (3D) space, $\{p_1, p_2, \cdots , p_N\}$. The problem ...
Cal-linux's user avatar
  • 145
0 votes
0 answers
122 views

Eigenvectors, Singular Vectors, and Excel

First time asker here. First off, I know I should be doing this in R or Python. I will. For now I'm reading a textbook, using simple examples and Excel to try to learn the concepts of linear algebra. ...
jlewis's user avatar
  • 13
2 votes
0 answers
47 views

How to find two small matrices $M_1$ and $M_2$ such that $M_1 M_2 A \approx M A$?

If we have a matrix $M$ and we want to find its least squares approximation as the product of two smaller (as in less rows or columns) matrices $M_1M_2$ of a given size, we can simply run SVD and pick ...
BluerFrog's user avatar
1 vote
1 answer
108 views

Clarifying the constraints used in deriving the Principal Components of PCA

In studying principal components analysis, I am confused by one point. For a set of $N$ (zero-centered) data points of dimension $m$, projected to a dimension $k < m$, we want a set of vectors of ...
IntegrateThis's user avatar
1 vote
0 answers
98 views

Understanding the solution to the varimax rotation problem

I'd like to preface this post by saying that this is my first post on stack exchange, so if there is anything to improve, be it redaction or just the structuring of posts, I'm more than willing to ...
dizzyAfterMatrixRotations's user avatar
1 vote
0 answers
30 views

Can you use EVD, SVD, PCA to solve least squares? (Intuitive Understanding)

I am trying to fully understand/demystify EVD, SVD and PCA. Am I right to assume all of these are tools/methods to solve least squares (not only but for CG)? If I am not wrong, even there are multiple ...
Joan Venge's user avatar
2 votes
0 answers
29 views

strategies for looking at the phase space of a system with 6 dimensions

I have a system of odes where the state vector has 6 elements. The system is a population biology model, where I am tracking the evolution of some competing species over time. Now I was trying to ...
krishnab's user avatar
  • 2,341
1 vote
0 answers
381 views

Computation of exponential and logarithmic maps on Riemann manifolds

In my computational problem, I have a Riemann submanifold $S^{1000}$ embedded in $\mathbb{R}^{300000}$. I can numerically compute the induced metric tensor and the Jacobian. I have no analytical ...
Luke's user avatar
  • 11
1 vote
1 answer
255 views

Proof of the variance of one-dimensional projections

In Bishop's Pattern Recognition And Machine Learning book, Chapter 12, Suppose $X$ is an uncentered data matrix and $\bar{x}=\frac{1}{m}\sum_ix_i$ is the sample mean of the columns of $X$. For the ...
Jayden Rice's user avatar
0 votes
1 answer
122 views

Where in PCA does the non-uniqueness of eigenvectors come from?

I tried comparing sklearn.decomposition.KernelPCA with a linear kernel to sklearn.decomposition.PCA on the same data set and got different eigenvectors. My understanding is that these should be ...
Galen's user avatar
  • 1,816
0 votes
1 answer
81 views

Proof of "The sum of squared distances from the points to the line is a minimum"

I'm reading "Introduction to linear algebra" of Gillbert Strang. PCA by SVD section. Text says that the sum of squared distances from the points to the line is a minimum and author is trying ...
Vanconts's user avatar
  • 149
3 votes
1 answer
99 views

quasi-PCA reconstruction of the matrix by orthogonal basis

let's say I have a "data" matrix $X$ of $N$ rows and $p$ cols with $N \gg p$. Now PCA with $L$ components can be formulated as $$X_L = argmin_{Y:rank(Y) = L} ||X- Y||^2_F $$, where Y is ...
Vladimir Kirilin's user avatar
3 votes
1 answer
194 views

How to compute principal components for a curvature found given XYZ points?

I have a certain XYZ set of points that make up an object. I chose a random point and make the nearest radius analysis and find the neighbors. From these neighbors, I get the green pointcloud curve ...
Hamzalihi's user avatar
0 votes
1 answer
820 views

projection of the data along the 1st k principal components

I'm a final year maths undergrad doing a course in multivariate data analysis, but I'm really struggling with the linear algebra. In particular the “projection of the data along the 1st k principal ...
d89J_kl's user avatar
  • 39
1 vote
0 answers
24 views

PCA factor $\frac{1}{n-1}$

I've seen multiple PCA derivations where the $\frac{1}{n}$ (for variance) or $\frac{1}{n-1}$ (for sample variance) is just omitted, e.g. here. I see that they are proportional to each other but is ...
jonithani123's user avatar
2 votes
0 answers
28 views

How to learn data points by minimizing a loss function given their pairwise distance matrix?

Suppose $x_i \in\mathbb{R}^2$ for $i=1,2,...9$ are unknown. I'm given the pair-wise distance matrix between these points $D$ which is a $9*9$ symmetric matrix. I want to learn these data points by ...
chesslad's user avatar
  • 2,533
0 votes
2 answers
701 views

What does singular value decomposition of covariance matrix represent?

I am reading the paper "Understanding dimensional collapse in contrastive self-supervised learning." The authors identified a dimensional collapse phenomenon: i.e. some dimension of ...
Noel's user avatar
  • 1
1 vote
1 answer
76 views

How to prove some insights regarding a new pca coordinate system

I've a question regarding pca variant. Let $X ∈ \Bbb R^{D×n}$ be a data matrix, $\{u_i\}^{d}i=1$ be the $d$ principal components of $X$, and where $μ ∈ \Bbb R^d$ is the sample mean vector and $1_n ∈ \...
joen joe's user avatar
0 votes
0 answers
48 views

Distance metric that is insensitive to correlated variables

I'm trying to find a suitable pairwise distance metric where the addition of correlated vectors results in (essentially) no change in the distance. Specifically, consider a set of $k$ vectors each of ...
user3037237's user avatar
2 votes
1 answer
215 views

How do you measure the 'explained variance' of arbitrary linear embeddings?

Question Elaboration: When I say 'linear embeddings' I mean a lower-dimensional representation of variables resulting from an arbitrary linear transformation. And when I say 'explained variance' I ...
profPlum's user avatar
  • 337
1 vote
0 answers
57 views

Kernel PCA similarity matrix analogy

The standard explanation to linear PCA begins with the covariance matrix. That is, for a dataset $D$ of dimension $N \times d$, the covariance matrix is given as $\sum = \frac{D^{T}D}{N}$ where the ...
Mathematicing's user avatar
0 votes
0 answers
19 views

Principal Component Analysis - How do these two representations not contradict each other?

enter image description here The representation here basically says each observation in a data set is a sum of products of some Zs and loadings. By Zs, I mean the component scores. Loadings are the ...
Soham Deshmukh's user avatar