How is it determined that the Covariance matrix has Eigenvectors which are in the direction of largest variation of a data set?
I suppose to derive this you would maybe use regression line for the normalised data set, after finding the regression line we have found the vector of largest variation. From the regression line vector we can find the orthogonal vector.
Are the vectors chosen to be Eigenvectors? How are the Eigen values found? So we choose these vectors in which they are Eigen vectors of a matrix A and this matrix A happens to be the covariance matrix?
Im basically missing the connection between choosing vectors in direction with greatest variance for a dataset and how these were determined to be related to the Covariance matrix. Because if you choose 3 Eigenvectors and find the matrix A essentially it will be random numbers, so does it turn out that these random numbers are covariances and variances?