Skip to main content

Questions tagged [principal-component-analysis]

Principal component analysis (PCA) is a linear dimensionality reduction technique. It reduces a multivariate dataset to a smaller set of constructed variables preserving as much information (as much variance) as possible. These variables, called principal components, are linear combinations of the input variables.

Filter by
Sorted by
Tagged with
-3 votes
0 answers
38 views

Check if 2 point clouds are the same up to coordinate flips and rotation

I want to check if 2 point clouds in N dimensions are the same up to rotations about the origin and coordinate swaps. I define a point cloud as a finite collection of points, already centered at the ...
batman247's user avatar
1 vote
1 answer
86 views

Prove that PCA decomposition captures all information in a factor model

Assume a data matrix $X \in \mathbb{R}^{N \times p_X}$. Let it have some exact lower dimensional factor representation $X = A F$, where $F \in \mathbb{R}^{N \times p_F}$ and $p_F < p_X$. Let the ...
fes's user avatar
  • 1,671
1 vote
0 answers
43 views

Proving Optimal Loss Analogous to PCA

Suppose we have a data distribution on $x ∈ R^d$. Suppose $E_x[x] = 0$, and let $Σ = E_x[xx^T]$ be the covariance of x. Let $Σ = USU^T$ be the spectral decomposition of Σ, with U orthonormal and S ...
MedMan27's user avatar
2 votes
1 answer
67 views

minimizing $| x x^T - A |^2$ for a covariance matrix $A$

For a given covariance matrix $A$ (so $A$ is symmetrical and positive semidefinite), I want to find a vector $x$ such that $| x x^T - A |^2$ is minimized. Here $|M|^2$ just means the sum of squares ...
CuriousMind's user avatar
  • 1,634
1 vote
1 answer
63 views

PCA Reconstruction Properties

Let $X \in \mathbb{R}^{n \times d}$ be our data matrix where $n$ is the number of examples and $d$ is the feature dimension. Applying PCA to $X$, we get a low-dimensional representation $A \in \mathbb{...
FountainTree's user avatar
1 vote
1 answer
39 views

Intuition behind Principal Component Analysis: linear combinations of original dimensions vs the PC direction as linear function of y and x

So in this example here the Principal Components (PC) are: $$ \begin{align} PC_1 (Size) &= 0.707 * Height + 0.707 * Diameter \quad (1)\\ PC_2 (Shape) &= 0.707 * Height - 0.707 * Diameter \...
yyyLLL's user avatar
  • 33
1 vote
0 answers
25 views

Question about a linear algebra detail of Kernel PCA

As is shown in this question kernel pca eigenproblem and many other refernce materials about kernel PCA. They all point out that the solution of $K^2a_j=\lambda_jnKa_j$ and $Ka_j=\lambda_jna_j$ only ...
Wang Pengxin's user avatar
0 votes
0 answers
23 views

Can principal components changed by a normalization method be used to construct original data shape with SVD

I'm planning to use an algorithm called Harmony, designed for data normalization, particularly in the context of single cell data analysis. Harmony operates by taking principal components (PCs) as ...
MadmanLee's user avatar
  • 101
0 votes
0 answers
38 views

Solving Matrix Equation using SVD

I'm reading this paper by Bishop and Tipping. They solve the equation $$(SC^{-1} - I)W = 0$$ Where $W \in \mathbb{R}^{d \times q}$ and $S , C \in \mathbb{R}^{d \times d}$ and $C = WW^T + \sigma^2 I$ ...
Harry's user avatar
  • 43
0 votes
0 answers
34 views

Eigenvectors and Eigenvalues of Matrix product with tall matrix with itself

Given tall matrix $B \in \mathbf{R}^{n \times r}$, where $n \gg r$, let $$ A := B B^T $$ and the columns of $B$ are in general not orthogonal to each other. I would like to get the eigenvectors and ...
checkThisOut's user avatar
0 votes
0 answers
32 views

PCA: derivation of successive principal axes

Let $\Sigma$ be the covariance matrix. The first principal axis of PCA, $u_1$, can be found by solving the optimization problem: $$\max_{u_{1}}=u^T_1\Sigma u_1$$ subject to the constraint $u^T_1u_1=1$....
s1m0n3's user avatar
  • 13
1 vote
1 answer
123 views

Does PCA always find the best-fitting plane?

Here, the best-fitting plane is the plane that minimizes the sum of squared perpendicular distance from the data points to the plane. In other words, the best-fitting plane is the solution to the ...
iklnot's user avatar
  • 13
0 votes
0 answers
28 views

Degrees of Freedom in PCA

Suppose we are doing PCA over a historic time series of temperatures. The feature for the PCA to be explained is the time when the temperature was observed every hour. Let’s say for the sake of ...
Identicon's user avatar
1 vote
0 answers
56 views

Confusion regarding the geometrical meaning of singular values in SVD

I am trying to visualize in MATLAB the relationship between the singular value decomposition (SVD) of a matrix of points. To simplify the problem, I am working in 2D and I am considering an ellipse ...
gwn's user avatar
  • 121
0 votes
0 answers
15 views

Principal Component Analysis Based on Scatter Matrices

There is a paper on robust PCA based on scatter matrices: https://wis.kuleuven.be/statdatascience/robust/papers/2005/hubertrousseeuwvandenbranden-robpca-technom-2005.pdf. PCA is generally performed ...
Lyricist's user avatar
1 vote
0 answers
25 views

Canonical Variables and maximization of Covariance [closed]

I've been asked to find the deterministic unitary vectors $a\in \mathbb{R}^p$ and $b \in \mathbb{R}^q$ that maximize $\Cov(a^TX,b^TY)$ for $X$ a $p$-dimensional random vector, and $Y$ a $q$-...
juanan's user avatar
  • 76
0 votes
0 answers
60 views

Proving the relationship between mean-squared error of 1D projection of PCA and the largest eigenvalue of the covariance matrix

I was studying the topic of principal component analysis and came across this problem that I was not able to prove. Consider a data matrix, X and its covariance matrix, S. I know that taking the ...
DataStatsExplorer's user avatar
0 votes
0 answers
18 views

Is this a useful dimensionality reduction on real projective space?

In the following, I will derive a dimensionality reduction for the real projective space. As I will use scalar product as a means of distance, I am unsure if the dimensionality reduction has a useful ...
Niklas Netter's user avatar
0 votes
1 answer
22 views

the first principal component’s variance

The expression comes from The Elements of Statistical Learningp66(3.49) Related part about this is shown in below figure. Where Using SVD, $X$ has the form: $X=UDV^T$. And $D$ is a diagonal matrix, ...
chenqile's user avatar
2 votes
0 answers
230 views

Proof of Eckart-Young Theorem (Mathematics for Machine Learning, Deisenroth)

I am trying to understand a proof of the Eckart-Young Theorem (source in title). Let me add the definition and proof they provided, afterward I'll say what I do not understand. Proof: Some points I ...
Aaron's user avatar
  • 67
1 vote
1 answer
288 views

Why do the "important" eigenvectors of a graph Laplacian have small-magnitude eigenvalues?

In spectral clustering, one computes sample-sample similarities, then from this computes a graph Laplacian matrix. (Typically, one uses the symmetrically normalized Laplacian matrix, but the pattern I'...
calmcc's user avatar
  • 261
0 votes
0 answers
39 views

Should a gram matrix be $R = Q^TQ$ or $R = QQ^T$?

I'm reading a paper about fisher faces, that using kernel PCA together with LDA. The probelm I have is to understand $\Phi(x)$, I know that is an arbitary function for the vector $x$. But I don't ...
euraad's user avatar
  • 2,962
0 votes
0 answers
54 views

How to interpret the diagonalization of a scatter matrix in Principal Component Analysis (PCA)?

Let $X \in \mathbb{R}^{2 \times n}$ be the data matrix, $S_n = (X - \hat{\mu})(X - \hat{\mu})^T$ be the scatter matrix, and $$ S_n = \begin{array}{cc} \begin{bmatrix} 3 & -4 \\ -4 & 3 \end{...
Salman Akbar's user avatar
0 votes
1 answer
137 views

Suppose that the variance-covariance matrix of a $p$-dimensional random vector $X$ is

Suppose that the variance-covariance matrix of a $p$ -dimensional random vector $X$ is $\Sigma=\sigma_{ij}$ for all $i,j=1, 2, ...,p$. Show that the coefficients of the first principal component have ...
RV math's user avatar
  • 67
0 votes
0 answers
48 views

Order of the vectors in principal component analysis

For a given matrix $A$, the Principle Component Analysis (PCA) is done by finding the eigenvalues/eigenvectors of the covariance matrix associated with $A$. However, the entries of the covariance ...
user2167741's user avatar
0 votes
0 answers
41 views

Why the number of non-zero singular values is $n-1$ for $X\in \mathbb{R}^{n \times m}$ when $n\ll m$

There's a statement in the paper Phatak (1997) In most spectroscopic problems, the number of non-zero singular values is $n-1$ for $X\in \mathbb{R}^{n \times m}$ when $n\ll m$. Consequently, at most $...
WeiShan Ng's user avatar
0 votes
0 answers
118 views

Proof that PCA is equivalent to MDS when using Euclidean distances

As I was watching a video explaining how MDS works, the narrator mentioned that PCA is equivalent to MDS when Euclidean distances are used. I got confused as to how that's the case. My guess is that ...
baoiba's user avatar
  • 121
1 vote
0 answers
25 views

Relation of principal component analysis between a matrix and its transopose

With a matrix $X_{n\times p}$ ($n>p$), we perform a principal component analysis: $T_{n\times p}=X_{n\times p}W_{p\times p}$ where $W$ is the loadings matrix while $T$ is the scores matrix for $X$. ...
bluepole's user avatar
  • 153
1 vote
1 answer
81 views

Can SVM be special case of PCA?

Let $X$ and $Y$ two linearly separable finite subsets of a $K$-dimensional real vector space $V$ with orthonormal basis $A = \{a_1,\ldots, a_K\}$. The covariance matrix $\Sigma_A$ of the set $X \cup Y$...
Alberto Carraro's user avatar
0 votes
0 answers
61 views

The coskewness and cokurtosis of uncorrelated standardized random vector

I was conducting the Karhunen-Loeve (K-L) Expansion for a random vector. Based on the KL expansion, I transformed the original random vector into a standardized random vector $\boldsymbol{X}=[X_1,X_2,\...
Alexander's user avatar
1 vote
1 answer
534 views

Why the singular values in SVD are always hierarchical/descending?

Please, I'm trying to understand why singular values (SV) are always hierarchical/descending. At the beginning of my studies, I thought that the hierarchy of sigmas ($ \sigma_1 \geq \sigma_2 \geq ... \...
Caio Rímoli's user avatar
0 votes
1 answer
28 views

Find line with respect to which the moment of inertia is minimized

Consider a function $F(x,y,z): \mathbb{R}^3 \mapsto \mathbb{R}^+$ (i.e., $F(x,y,z) > 0 ~~\forall~ x,y,z$) and consider a set of points in the (3D) space, $\{p_1, p_2, \cdots , p_N\}$. The problem ...
Cal-linux's user avatar
  • 144
0 votes
0 answers
150 views

Eigenvectors, Singular Vectors, and Excel

First time asker here. First off, I know I should be doing this in R or Python. I will. For now I'm reading a textbook, using simple examples and Excel to try to learn the concepts of linear algebra. ...
jlewis's user avatar
  • 13
2 votes
0 answers
47 views

How to find two small matrices $M_1$ and $M_2$ such that $M_1 M_2 A \approx M A$?

If we have a matrix $M$ and we want to find its least squares approximation as the product of two smaller (as in less rows or columns) matrices $M_1M_2$ of a given size, we can simply run SVD and pick ...
BluerFrog's user avatar
1 vote
1 answer
120 views

Clarifying the constraints used in deriving the Principal Components of PCA

In studying principal components analysis, I am confused by one point. For a set of $N$ (zero-centered) data points of dimension $m$, projected to a dimension $k < m$, we want a set of vectors of ...
IntegrateThis's user avatar
1 vote
0 answers
107 views

Understanding the solution to the varimax rotation problem

I'd like to preface this post by saying that this is my first post on stack exchange, so if there is anything to improve, be it redaction or just the structuring of posts, I'm more than willing to ...
dizzyAfterMatrixRotations's user avatar
1 vote
0 answers
30 views

Can you use EVD, SVD, PCA to solve least squares? (Intuitive Understanding)

I am trying to fully understand/demystify EVD, SVD and PCA. Am I right to assume all of these are tools/methods to solve least squares (not only but for CG)? If I am not wrong, even there are multiple ...
Joan Venge's user avatar
2 votes
0 answers
29 views

strategies for looking at the phase space of a system with 6 dimensions

I have a system of odes where the state vector has 6 elements. The system is a population biology model, where I am tracking the evolution of some competing species over time. Now I was trying to ...
krishnab's user avatar
  • 2,531
1 vote
0 answers
472 views

Computation of exponential and logarithmic maps on Riemann manifolds

In my computational problem, I have a Riemann submanifold $S^{1000}$ embedded in $\mathbb{R}^{300000}$. I can numerically compute the induced metric tensor and the Jacobian. I have no analytical ...
Luke's user avatar
  • 11
1 vote
1 answer
302 views

Proof of the variance of one-dimensional projections

In Bishop's Pattern Recognition And Machine Learning book, Chapter 12, Suppose $X$ is an uncentered data matrix and $\bar{x}=\frac{1}{m}\sum_ix_i$ is the sample mean of the columns of $X$. For the ...
Jayden Rice's user avatar
0 votes
1 answer
188 views

Where in PCA does the non-uniqueness of eigenvectors come from?

I tried comparing sklearn.decomposition.KernelPCA with a linear kernel to sklearn.decomposition.PCA on the same data set and got different eigenvectors. My understanding is that these should be ...
Galen's user avatar
  • 1,876
0 votes
1 answer
118 views

Proof of "The sum of squared distances from the points to the line is a minimum"

I'm reading "Introduction to linear algebra" of Gillbert Strang. PCA by SVD section. Text says that the sum of squared distances from the points to the line is a minimum and author is trying ...
Vanconts's user avatar
3 votes
1 answer
104 views

quasi-PCA reconstruction of the matrix by orthogonal basis

let's say I have a "data" matrix $X$ of $N$ rows and $p$ cols with $N \gg p$. Now PCA with $L$ components can be formulated as $$X_L = argmin_{Y:rank(Y) = L} ||X- Y||^2_F $$, where Y is ...
Vladimir Kirilin's user avatar
3 votes
1 answer
241 views

How to compute principal components for a curvature found given XYZ points?

I have a certain XYZ set of points that make up an object. I chose a random point and make the nearest radius analysis and find the neighbors. From these neighbors, I get the green pointcloud curve ...
Hamzalihi's user avatar
0 votes
1 answer
1k views

projection of the data along the 1st k principal components

I'm a final year maths undergrad doing a course in multivariate data analysis, but I'm really struggling with the linear algebra. In particular the “projection of the data along the 1st k principal ...
d89J_kl's user avatar
  • 39
1 vote
0 answers
26 views

PCA factor $\frac{1}{n-1}$

I've seen multiple PCA derivations where the $\frac{1}{n}$ (for variance) or $\frac{1}{n-1}$ (for sample variance) is just omitted, e.g. here. I see that they are proportional to each other but is ...
jonithani123's user avatar
2 votes
0 answers
28 views

How to learn data points by minimizing a loss function given their pairwise distance matrix?

Suppose $x_i \in\mathbb{R}^2$ for $i=1,2,...9$ are unknown. I'm given the pair-wise distance matrix between these points $D$ which is a $9*9$ symmetric matrix. I want to learn these data points by ...
chesslad's user avatar
  • 2,533
1 vote
2 answers
999 views

What does singular value decomposition of covariance matrix represent?

I am reading the paper "Understanding dimensional collapse in contrastive self-supervised learning." The authors identified a dimensional collapse phenomenon: i.e. some dimension of ...
Noel's user avatar
  • 11
1 vote
1 answer
80 views

How to prove some insights regarding a new pca coordinate system

I've a question regarding pca variant. Let $X ∈ \Bbb R^{D×n}$ be a data matrix, $\{u_i\}^{d}i=1$ be the $d$ principal components of $X$, and where $μ ∈ \Bbb R^d$ is the sample mean vector and $1_n ∈ \...
joen joe's user avatar
0 votes
0 answers
48 views

Distance metric that is insensitive to correlated variables

I'm trying to find a suitable pairwise distance metric where the addition of correlated vectors results in (essentially) no change in the distance. Specifically, consider a set of $k$ vectors each of ...
user3037237's user avatar