class: center, middle, inverse, title-slide .title[ # Principle Component Analysis(PCA) ] .author[ ### Lin Yu ] .date[ ### 2023-03-20 ] --- ## Background ``` ## [1] "correlation between X1 and X2: 0.9" ``` <img src="data:image/png;base64,#PCA_files/figure-html/unnamed-chunk-1-1.png" style="display: block; margin: auto;" /> --- class: middle,center, inverse #Intuition ##fitting a p-dimensional ellipsoid to the data by **stretching** and **rotating** the coordination system. --- ## Background ``` ## [1] "correlation between X1 and X2: 0.9" ``` <img src="data:image/png;base64,#PCA_files/figure-html/unnamed-chunk-2-1.png" style="display: block; margin: auto;" /> --- # Which is better? <img src="data:image/png;base64,#PCA_files/figure-html/unnamed-chunk-3-1.png" style="display: block; margin: auto;" /> --- ## Goal .pull-left[ maximize `\(|d_1|+|d_2|+|d_3|+...+|d_n|\)`, equivalent to maximize `\(d_1^2+d_2^2+d_3^2+...+d_n^2\)` Qeustion: how to represent `\(d\)`? $$ d_1 = \left( `\begin{matrix} a_1 \\ b_1 \\ \end{matrix}` \right) . \left( `\begin{matrix} e11 \\ e12 \\ \end{matrix}` \right) $$ where `\(a_1\)`, `\(b_1\)` is the coordinate of the measured data point; e11 and e12 are the element of the basis vector e1. ] .pull-right[ ![](data:image/png;base64,#PCA_files/figure-html/unnamed-chunk-4-1.png)<!-- --> ] --- ## working example <img src="data:image/png;base64,#PCA_files/figure-html/unnamed-chunk-5-1.png" style="display: block; margin: auto;" /> --- ##(cont'd) suppose we have two data points, the goal can be written as: `\(d_1^2+d_2^2 = (a_1*e11+b_1*e12)^2 +(a_2*e11+b_2*e12)^2\)` .pull-left[ $$ = \left( `\begin{matrix} e11 & e12\\ \end{matrix}` \right) . \left( `\begin{matrix} a_1^2+a_2^2 & a_1*b_1+a_2*b_2 \\ a_1*b_1+a_2*b_2 & b_1^2+b_1^2 \\ \end{matrix}` \right) . \left( `\begin{matrix} e11 \\ e12 \\ \end{matrix}` \right) $$ `$$= e^T . U \Sigma U^T . e =( e^T . U) \Sigma ( U^T . e) = n^T \Sigma n \text{ (known as SVD)}$$` $$ = n^T \left( `\begin{matrix} \sigma_1 & 0 \\ 0 & \sigma_2 \\ \end{matrix}` \right) n = \left( `\begin{matrix} n_1 & n_2 \\ \end{matrix}` \right) \left( `\begin{matrix} \sigma_1 & 0 \\ 0 & \sigma_2 \\ \end{matrix}` \right) \left( `\begin{matrix} n_1 \\ n_2 \\ \end{matrix}` \right) $$ $$ = n_1^2 \times\sigma_1 + n_2^2 \times \sigma_2 \text{ => where } n_1^2+n_2^2=1 \text{ and } \sigma_1 > \sigma_2$$ when `\(n_1 = 0\)`, `\(n_2 = 0\)`, `\(d_1^2+d_2^2\)` max `$$( U^T . e) = n, e = Un, \text{where } n \text{ is } (1,0)^T$$` ] --- <img src="data:image/png;base64,#PCA_files/figure-html/unnamed-chunk-6-1.png" style="display: block; margin: auto;" /> $$= e^T . U \Sigma U^T . e =( e^T . U) \Sigma ( U^T . e) = n^T \Sigma n $$ --- #Steps: 1. Center the values of each variable in the dataset on 0 (by subtracting the mean of the variable's observed values from each of those values) 2. Compute the covariance matrix of the data and calculate the eigenvalues and corresponding eigenvectors of this covariance matrix 3. Normalize each of the orthogonal eigenvectors to turn them into unit vectors --- class:inverse,middle,center #Visualization --- ##The scree plot .center[ ![](data:image/png;base64,#figure/PCA_scree.png) ] .center[(help choose how many PCs to retain)] --- #The profile plot <img src="data:image/png;base64,#figure/PCA_profile.png" width="853" height="450" /> - shows the correlations between each PC and the original variables - will become very crowded and hard to read when have many variables --- #The score plot <img src="data:image/png;base64,#figure/PCA_score2.png" width="853" height="500" /> .center[projection of data onto the span of PC1 and PC2] --- #The loadings plots <img src="data:image/png;base64,#figure/PCA_loadings.png" width="853" height="500" /> .center[shows the relationship between the PCs and the original variables] --- #The biplot .pull-left[ overlays a score plot and a loadings plot in a single graph - The cosine of the angle between a vector and an axis indicates the importance of the **contribution of the corresponding variable to the principal component**. - The cosine of the angle between pairs of vectors indicates **correlation between the corresponding variables**. Highly correlated variables point in similar directions; - Points that are close to each other in the biplot represent observations with similar values. ] .pull-right[ <img src="data:image/png;base64,#figure/biplotJK.png" width="500" height="500" /> ] --- class:inverse, middle,center # Application --- #1. Image compression .pull-left[ ``` ## (156, 194, 3) ``` <img src="data:image/png;base64,#figure/opengenus_logo.jpg" width="300" height="300" style="display: block; margin: auto auto auto 0;" /> ] .pull-right[ ``` ## (156, 194, 3) ``` <img src="data:image/png;base64,#PCA_files/figure-html/unnamed-chunk-14-1.png" width="672" /> ] --- class:inverse, middle,center #2. Face recognition
Download pcaface.pdf
--- #References [1] https://blogs.sas.com/content/iml/2019/11/06/what-are-biplots.html#:~:text=A%20biplot%20is%20an%20overlay%20of%20a%20score,plots%20and%20overlay%20them%20on%20a%20single%20plot. [2] https://www.zhihu.com/question/41120789/answer/481966094 [3] https://en.wikipedia.org/wiki/Principal_component_analysis#Intuition