# Questions tagged [principal-component-analysis]

Principal component analysis (PCA) is a linear dimensionality reduction technique. It reduces a multivariate dataset to a smaller set of constructed variables preserving as much information (as much variance) as possible. These variables, called principal components, are linear combinations of the input variables.

124 questions
Filter by
Sorted by
Tagged with
1 vote
68 views

### Can SVM be special case of PCA?

Let $X$ and $Y$ two linearly separable finite subsets of a $K$-dimensional real vector space $V$ with orthonormal basis $A = \{a_1,\ldots, a_K\}$. The covariance matrix $\Sigma_A$ of the set $X \cup Y$...
16 views

10 views

### Determining the direction of greatest change in variance between two samples

Say we have two samples in the same multivariate space. We can perform some analysis like Principal Component Analysis to determine the directions of greatest variance for each of the two samples. But ...
32 views

### Relationship between the discrete cosine transform and the Karhunen–Loève transform

The Wikipedia article on Discrete cosine transform states: For strongly correlated Markov processes, the DCT can approach the compaction efficiency of the Karhunen-Loève transform (which is optimal ...
• 389
21 views

### Find line with respect to which the moment of inertia is minimized

Consider a function $F(x,y,z): \mathbb{R}^3 \mapsto \mathbb{R}^+$ (i.e., $F(x,y,z) > 0 ~~\forall~ x,y,z$) and consider a set of points in the (3D) space, $\{p_1, p_2, \cdots , p_N\}$. The problem ...
• 135
9 views

### Is it possible to combine (or update) principal component (PC) from two different matrices?

Let's say I have two real matrices, $A$ and $B$, size of $m$ by $n$. I can concat those matrices horizontally which results in matrix $C$, which has size of $m$ by $2n$. So my question is the ...
• 23
17 views

### How to prove the principal components minimise the least squares?

Consider the factor model: $$\bf x_t =\Lambda f_t + e_t$$ where $\bf x_t$ is an $n\times 1$ vector; $\bf f_t$ are $r\times 1$ factors; $\bf \Lambda$ is the factor loading matrix. We want to find ...
• 3
52 views

### Eigenvectors, Singular Vectors, and Excel

First time asker here. First off, I know I should be doing this in R or Python. I will. For now I'm reading a textbook, using simple examples and Excel to try to learn the concepts of linear algebra. ...
41 views

### How to find two small matrices $M_1$ and $M_2$ such that $M_1 M_2 A \approx M A$?

If we have a matrix $M$ and we want to find its least squares approximation as the product of two smaller (as in less rows or columns) matrices $M_1M_2$ of a given size, we can simply run SVD and pick ...
1 vote
63 views

### Clarifying the constraints used in deriving the Principal Components of PCA

In studying principal components analysis, I am confused by one point. For a set of $N$ (zero-centered) data points of dimension $m$, projected to a dimension $k < m$, we want a set of vectors of ...
• 3,778
61 views

### Understanding the solution to the varimax rotation problem

I'd like to preface this post by saying that this is my first post on stack exchange, so if there is anything to improve, be it redaction or just the structuring of posts, I'm more than willing to ...
1 vote
15 views

### Can you use EVD, SVD, PCA to solve least squares? (Intuitive Understanding)

I am trying to fully understand/demystify EVD, SVD and PCA. Am I right to assume all of these are tools/methods to solve least squares (not only but for CG)? If I am not wrong, even there are multiple ...
• 111
25 views

### strategies for looking at the phase space of a system with 6 dimensions

I have a system of odes where the state vector has 6 elements. The system is a population biology model, where I am tracking the evolution of some competing species over time. Now I was trying to ...
• 1,873
1 vote
150 views

### Computation of exponential and logarithmic maps on Riemann manifolds

In my computational problem, I have a Riemann submanifold $S^{1000}$ embedded in $\mathbb{R}^{300000}$. I can numerically compute the induced metric tensor and the Jacobian. I have no analytical ...
• 11
1 vote
124 views

### Proof of the variance of one-dimensional projections

In Bishop's Pattern Recognition And Machine Learning book, Chapter 12, Suppose $X$ is an uncentered data matrix and $\bar{x}=\frac{1}{m}\sum_ix_i$ is the sample mean of the columns of $X$. For the ...
• 303
47 views

### Where in PCA does the non-uniqueness of eigenvectors come from?

I tried comparing sklearn.decomposition.KernelPCA with a linear kernel to sklearn.decomposition.PCA on the same data set and got different eigenvectors. My understanding is that these should be ...
• 1,687
46 views

### Proof of "The sum of squared distances from the points to the line is a minimum"

I'm reading "Introduction to linear algebra" of Gillbert Strang. PCA by SVD section. Text says that the sum of squared distances from the points to the line is a minimum and author is trying ...
• 113
16 views

### Solving for/Generating Coefficients to Reconstruct a Vector Using Defined Basis

I am working on a project in MATLAB (involving principle component analysis) where I have a generated a set of basis vectors using a test set, and am attempting to reconstruct a vector outside the ...
• 1
26 views

### PCA on a data matrix with features on a different scale

Suppose I have a data matrix $X$ of dimensions [100, 5] that describes 100 different people each with a series of 5 features that define them. Let these 5 features be, for example, weight (kg), height ...
• 25
90 views

### quasi-PCA reconstruction of the matrix by orthogonal basis

let's say I have a "data" matrix $X$ of $N$ rows and $p$ cols with $N \gg p$. Now PCA with $L$ components can be formulated as $$X_L = argmin_{Y:rank(Y) = L} ||X- Y||^2_F$$, where Y is ...
33 views

1 vote
36 views

### Help with a PCA procedure.

I'm studying THIS paper which builds an index from a set of observed variables using Principal Component Analysis (PCA). However, the procedure described in Section 3.1 by formulas (6)-(8) confused me....
• 2,545
89 views

### What is the correct formula for the covariance matrix?

I am solving about principal component analysis (PCA) and I stumbled upon a place where I need to calculate the covariance matrix, I am seeing varieties of formula. Here are some that I have found: ...
• 21
1 vote
14 views

### How to create separation of "sides"/signs in PCA / eigenvector "directionality of data" analysis?

How to create separation of "sides"/signs in PCA / eigenvector "directionality of data" analysis? Since PCA will only give eigenvectors that show the principal axes, but it does ...
• 7,042
1 vote
In the topic of Principal component analysis in high dimensions I have given the following task: Let $X\in\mathbb R^{n\times d}$ and $w\sim\mathcal N(0,\sigma^2I_n)$. Show that for any $\lambda>0$ ...