Questions tagged [principal-component-analysis]
Principal component analysis (PCA) is a linear dimensionality reduction technique. It reduces a multivariate dataset to a smaller set of constructed variables preserving as much information (as much variance) as possible. These variables, called principal components, are linear combinations of the input variables.
124
questions
1
vote
1
answer
68
views
Can SVM be special case of PCA?
Let $X$ and $Y$ two linearly separable finite subsets of a $K$-dimensional real vector space $V$ with orthonormal basis $A = \{a_1,\ldots, a_K\}$. The covariance matrix $\Sigma_A$ of the set $X \cup Y$...
0
votes
0
answers
16
views
The coskewness and cokurtosis of uncorrelated standardized random vector
I was conducting the Karhunen-Loeve (K-L) Expansion for a random vector.
Based on the KL expansion, I transformed the original random vector into a standardized random vector $\boldsymbol{X}=[X_1,X_2,\...
- 63
1
vote
1
answer
79
views
Why the singular values in SVD are always hierarchical/descending?
Please, I'm trying to understand why singular values (SV) are always hierarchical/descending. At the beginning of my studies, I thought that the hierarchy of sigmas ($ \sigma_1 \geq \sigma_2 \geq ... \...
- 19
0
votes
0
answers
10
views
Determining the direction of greatest change in variance between two samples
Say we have two samples in the same multivariate space. We can perform some analysis like Principal Component Analysis to determine the directions of greatest variance for each of the two samples. But ...
- 1
0
votes
0
answers
32
views
Relationship between the discrete cosine transform and the Karhunen–Loève transform
The Wikipedia article on Discrete cosine transform states:
For strongly correlated Markov processes, the DCT can approach the compaction efficiency of the Karhunen-Loève transform (which is optimal ...
- 389
0
votes
1
answer
21
views
Find line with respect to which the moment of inertia is minimized
Consider a function $F(x,y,z): \mathbb{R}^3 \mapsto \mathbb{R}^+$ (i.e., $F(x,y,z) > 0 ~~\forall~ x,y,z$) and consider a set of points in the (3D) space, $\{p_1, p_2, \cdots , p_N\}$.
The problem ...
- 135
0
votes
0
answers
9
views
Is it possible to combine (or update) principal component (PC) from two different matrices?
Let's say I have two real matrices, $A$ and $B$, size of $m$ by $n$.
I can concat those matrices horizontally which results in matrix $C$, which has size of $m$ by $2n$.
So my question is the ...
- 23
0
votes
0
answers
17
views
How to prove the principal components minimise the least squares?
Consider the factor model:
$$ \bf x_t =\Lambda f_t + e_t $$
where $\bf x_t$ is an $n\times 1$ vector; $\bf f_t$ are $r\times 1$ factors; $\bf \Lambda$ is the factor loading matrix.
We want to find ...
- 3
0
votes
0
answers
52
views
Eigenvectors, Singular Vectors, and Excel
First time asker here.
First off, I know I should be doing this in R or Python. I will. For now I'm reading a textbook, using simple examples and Excel to try to learn the concepts of linear algebra. ...
- 1
2
votes
0
answers
41
views
How to find two small matrices $M_1$ and $M_2$ such that $M_1 M_2 A \approx M A$?
If we have a matrix $M$ and we want to find its least squares approximation as the product of two smaller (as in less rows or columns) matrices $M_1M_2$ of a given size, we can simply run SVD and pick ...
- 21
1
vote
1
answer
63
views
Clarifying the constraints used in deriving the Principal Components of PCA
In studying principal components analysis, I am confused by one point.
For a set of $N$ (zero-centered) data points of dimension $m$, projected to a dimension $k < m$, we want a set of vectors of ...
- 3,778
0
votes
0
answers
61
views
Understanding the solution to the varimax rotation problem
I'd like to preface this post by saying that this is my first post on stack exchange, so if there is anything to improve, be it redaction or just the structuring of posts, I'm more than willing to ...
1
vote
0
answers
15
views
Can you use EVD, SVD, PCA to solve least squares? (Intuitive Understanding)
I am trying to fully understand/demystify EVD, SVD and PCA. Am I right to assume all of these are tools/methods to solve least squares (not only but for CG)? If I am not wrong, even there are multiple ...
- 111
2
votes
0
answers
25
views
strategies for looking at the phase space of a system with 6 dimensions
I have a system of odes where the state vector has 6 elements. The system is a population biology model, where I am tracking the evolution of some competing species over time.
Now I was trying to ...
- 1,873
1
vote
0
answers
150
views
Computation of exponential and logarithmic maps on Riemann manifolds
In my computational problem, I have a Riemann submanifold $S^{1000}$ embedded in $\mathbb{R}^{300000}$. I can numerically compute the induced metric tensor and the Jacobian. I have no analytical ...
- 11
1
vote
1
answer
124
views
Proof of the variance of one-dimensional projections
In Bishop's Pattern Recognition And Machine Learning book, Chapter 12,
Suppose $X$ is an uncentered data matrix and $\bar{x}=\frac{1}{m}\sum_ix_i$ is the sample mean of the columns of $X$.
For the ...
- 303
0
votes
1
answer
47
views
Where in PCA does the non-uniqueness of eigenvectors come from?
I tried comparing sklearn.decomposition.KernelPCA with a linear kernel to sklearn.decomposition.PCA on the same data set and got different eigenvectors. My understanding is that these should be ...
- 1,687
0
votes
1
answer
46
views
Proof of "The sum of squared distances from the points to the line is a minimum"
I'm reading "Introduction to linear algebra" of Gillbert Strang.
PCA by SVD section.
Text says that the sum of squared distances from the points to the line is a minimum and author is trying ...
- 113
0
votes
0
answers
16
views
Solving for/Generating Coefficients to Reconstruct a Vector Using Defined Basis
I am working on a project in MATLAB (involving principle component analysis) where I have a generated a set of basis vectors using a test set, and am attempting to reconstruct a vector outside the ...
- 1
0
votes
0
answers
26
views
PCA on a data matrix with features on a different scale
Suppose I have a data matrix $X$ of dimensions [100, 5] that describes 100 different people each with a series of 5 features that define them. Let these 5 features be, for example, weight (kg), height ...
- 25
3
votes
1
answer
90
views
quasi-PCA reconstruction of the matrix by orthogonal basis
let's say I have a "data" matrix $X$ of $N$ rows and $p$ cols with $N \gg p$. Now PCA with $L$ components can be formulated as $$X_L = argmin_{Y:rank(Y) = L} ||X- Y||^2_F $$, where Y is ...
0
votes
0
answers
33
views
Sparse PCA algorithm clarification
The Sparse PCA algorithm proposed in this paper proposes two ways to calculate sparse principal components, first by using $$\hat \beta = \text{arg min}_\beta ||Z_i - X\beta||^2 + \lambda ||\beta||^2 +...
- 183
3
votes
1
answer
83
views
How to compute principal components for a curvature found given XYZ points?
I have a certain XYZ set of points that make up an object. I chose a random point and make the nearest radius analysis and find the neighbors. From these neighbors, I get the green pointcloud curve ...
- 73
0
votes
1
answer
197
views
projection of the data along the 1st k principal components
I'm a final year maths undergrad doing a course in multivariate data analysis, but I'm really struggling with the linear algebra. In particular the “projection of the data along the 1st k principal ...
- 39
1
vote
0
answers
24
views
PCA factor $\frac{1}{n-1}$
I've seen multiple PCA derivations where the $\frac{1}{n}$ (for variance) or $\frac{1}{n-1}$ (for sample variance) is just omitted, e.g. here. I see that they are proportional to each other but is ...
- 183
2
votes
0
answers
28
views
How to learn data points by minimizing a loss function given their pairwise distance matrix?
Suppose $x_i \in\mathbb{R}^2$ for $i=1,2,...9$ are unknown. I'm given the pair-wise distance matrix between these points $D$ which is a $9*9$ symmetric matrix. I want to learn these data points by ...
- 2,371
0
votes
1
answer
165
views
What does singular value decomposition of covariance matrix represent?
I am reading the paper "Understanding dimensional collapse in contrastive self-supervised learning." The authors identified a dimensional collapse phenomenon:
i.e. some dimension of ...
- 1
0
votes
0
answers
20
views
Correlation on principal component analysis[PCA]
I am analysing a dataset on NBA players, using R. I am having some doubts on correlation in pca. I know the eigenvalues represent the variability in each PCA. Each eigenvalue has a eigenvector. I ...
- 33
1
vote
1
answer
66
views
How to prove some insights regarding a new pca coordinate system
I've a question regarding pca variant.
Let $X ∈ \Bbb R^{D×n}$ be a data matrix, $\{u_i\}^{d}i=1$ be the $d$ principal components of $X$, and where $μ ∈ \Bbb R^d$ is the sample mean vector and $1_n ∈ \...
- 19
0
votes
0
answers
45
views
Distance metric that is insensitive to correlated variables
I'm trying to find a suitable pairwise distance metric where the addition of correlated vectors results in (essentially) no change in the distance.
Specifically, consider a set of $k$ vectors each of ...
- 215
2
votes
1
answer
107
views
How do you measure the 'explained variance' of arbitrary linear embeddings?
Question Elaboration:
When I say 'linear embeddings' I mean a lower-dimensional representation of variables resulting from an arbitrary linear transformation. And when I say 'explained variance' I ...
- 317
1
vote
0
answers
51
views
Kernel PCA similarity matrix analogy
The standard explanation to linear PCA begins with the covariance matrix.
That is, for a dataset $D$ of dimension $N \times d$, the covariance matrix is given as $\sum = \frac{D^{T}D}{N}$ where the ...
- 5,957
0
votes
0
answers
18
views
Principal Component Analysis - How do these two representations not contradict each other?
enter image description here
The representation here basically says each observation in a data set is a sum of products of some Zs and loadings. By Zs, I mean the component scores. Loadings are the ...
2
votes
0
answers
57
views
Confused about the relationship between PCA and robust PCA
I recently learned about PCA and robust PCA. I understand that PCA is identifying the principal components by finding the eigenvectors of the covariance matrix (which of course contains information ...
- 31
1
vote
1
answer
47
views
Why is it possible to write the following term when using the summation solution to PCA?
I am going through PCA notes and I am stumped by the last line in the term expansion (see image below). How did we get $-2\sum_{j=1}^N\sum_{i=1}^k\alpha_{ji}x_j^Tv_i$ from the previous two middle ...
- 121
0
votes
1
answer
292
views
compute eigendecomposition to find eigenvalues and eigenvector
So I have matrix A which has (m×d) dimension and X = A.A^T
My X has d non-zero distinct eigenvalues and m > d
Usually I compute eigendecomposition matrix which has (mxm) dimension to find ...
3
votes
0
answers
50
views
unit vector after transformation with longer length than the eigenvector with highest eigenvalue
my question is a general one, can a unit vector after transformation have the length longer than length of the eigenvector which highest eigenvalue? is there any proof?
purpose is that I want to know ...
- 131
0
votes
1
answer
71
views
Singular Vector Decomposition and PCA interpretation
The covariance matrix of any data X -> N * D (N samples and D dimension ) would be Cov(X) = E[(X - E[X])(X - E[X])T]. Let's Assume (X - E[X])=Y. Thus Cov(X) = E[YYT]. Now from SVD we know that U ...
- 33
1
vote
0
answers
94
views
In PCA, how are maximising variance and minimising reconstruction error equivalent?
I understand that, given a normalised data matrix $X$, the PCA algorithm can be formulated as maximisation problem where objective function is:
$$ \max_{{\omega}} \omega'X'X\omega \quad\quad \text{ ...
1
vote
0
answers
36
views
Help with a PCA procedure.
I'm studying THIS paper which builds an index from a set of observed variables using Principal Component Analysis (PCA).
However, the procedure described in Section 3.1 by formulas (6)-(8) confused me....
- 2,545
2
votes
1
answer
89
views
What is the correct formula for the covariance matrix?
I am solving about principal component analysis (PCA) and I stumbled upon a place where I need to calculate the covariance matrix, I am seeing varieties of formula. Here are some that I have found:
...
- 21
1
vote
0
answers
14
views
How to create separation of "sides"/signs in PCA / eigenvector "directionality of data" analysis?
How to create separation of "sides"/signs in PCA / eigenvector "directionality of data" analysis?
Since PCA will only give eigenvectors that show the principal axes, but it does ...
- 7,042
1
vote
0
answers
19
views
Bound for a random matrix and a normal distributed random vector
In the topic of Principal component analysis in high dimensions I have given the following task:
Let $X\in\mathbb R^{n\times d}$ and $w\sim\mathcal N(0,\sigma^2I_n)$.
Show that for any $\lambda>0$ ...
1
vote
0
answers
20
views
Alternative definition of principal component analysis
I'm reading another definition of $PCA$, quite different from what I've been used to see and I'm a little bit confused:
given a dataset $X \in \mathbb{R}^{d \times m}$ that we approximate as $X \sim V^...
- 1,812
0
votes
1
answer
115
views
What does SCoTLASS stand for [closed]
To obtain sparse principal components, there's a technique called SCoLASS. Does anyone know what this acronym stands for? I suppose LASS stands for lasso, as it puts a constraint on an l_1-norm, but ...
- 13
0
votes
0
answers
62
views
Covariance matrix in PCA approach
While reading the brief discussion on the PCA approach in the book Deep Learning (Ian Goodfellow and Yoshua Bengio and Aaron Courville), I could not understand the passage shown in the figure.
...
- 25
1
vote
0
answers
70
views
PCA-algorithm dependence on number of principal components.
Consider the following idea: we have $(X,Y) \in \mathbb{R}^{n \times (d+k)}$ - initial data. Let's call $\bar{X} = \dfrac{1}{n}\sum_i x_i$ (the same for $\bar{Y}$). We want to predict $y$ by $x$ using ...
- 6,298
0
votes
1
answer
38
views
Show that $\frac{\sum^k_{i=1}\text{Var}(Y_i)}{\sum^p_{i=1}\text{Var}(X_i)}=\frac{\sum^k_{i=1}\lambda_i}{\sum^p_{i=1}\lambda_i}$
Let $X=(X_1,...,X_p)$ be a random vector with $\mathbb{E}(X)=\mu$ and covariance matrix $\text{Cov}(X)=\Sigma$. Define $Y=(Y_1,...,Y_p)^T=\mathbb{E}^TX$, where $E$ is a $p\times p$ matrix with columns ...
user895986
1
vote
1
answer
57
views
Projecting onto the space of Upper-Triangular-ish Matrices
I want to solve the independent component analysis (ICA) problem for a single time domain channel of input. The problem is, in my formulation, the mixing matrix has a special structure. I want to know ...
- 1,039
1
vote
0
answers
76
views
How to prove that eig($A_1$)+eig($A_2$)=eig($A'$)?
Given $3$ matrices:
$$\begin{aligned} A_1 &= \frac12 \left( X'X'^T + CX'X'^T + X'X'^TC + CX'X'^TC \right)\\ A_2 &= \frac12 \left( X'X'^T - CX'X'^T - X'X'^TC + CX'X'^TC \right)\\ A' &= X'X'^...
- 11