Question 5 of 5: Effectiveness of Principal Component Analysis

Question 5: Effectiveness of Principal Component Analysis (20 marks)

Question text not reproduced: the examination questions are © Engineers and Geoscientists BC. Open the official past paper (linked at the top of this page) to read the question, then follow the worked solution below.

Given. Two two-band variance–covariance matrices — Group A and Group B — whose diagonals are the band variances and whose off-diagonal is the inter-band covariance:

Find. Why PCA was effective for Group A but of little use for Group B — with a quantitative justification (correlation, eigenvalues, and the fraction of variance carried by the first principal component).

Approach. PCA rotates the two band axes onto the eigenvectors of the covariance matrix; the eigenvalues are the variances of the resulting principal components. PCA is effective — i.e. it reduces two bands to essentially one — only when the bands are highly correlated, so the first eigenvalue dominates and PC1 alone explains most of the total variance (the trace). For a symmetric $2\times2$ matrix the eigenvalues are available in closed form.

1. Inter-band correlation. Normalize each covariance by the geometric mean of the two variances, $r=\sigma_{12}/\sqrt{\sigma_{11}\sigma_{22}}$: $$r_A=\frac{4.5}{\sqrt{5.4\times 6.1}}=0.784,\qquad r_B=\frac{4.2}{\sqrt{28.0\times 16.4}}=0.196.$$ Group A's two bands are strongly correlated (redundant); Group B's are almost independent. Correlation is exactly the redundancy PCA exploits, so this already predicts the outcome.
2. Total variance (trace). The trace is invariant under the PCA rotation: $$\operatorname{tr}\Sigma_A = 5.4+6.1 = 11.5,\qquad \operatorname{tr}\Sigma_B = 28.0+16.4 = 44.4.$$
3. Eigenvalues (PC variances). For $\begin{bmatrix}a&b\\ b&d\end{bmatrix}$, $\lambda=\tfrac{a+d}{2}\pm\sqrt{\big(\tfrac{a-d}{2}\big)^2+b^2}$. Group A: $$\lambda=5.75\pm\sqrt{0.35^2+4.5^2}=5.75\pm 4.514 \;\Rightarrow\; \lambda_1=\boxed{10.26},\;\lambda_2=1.24.$$ Group B: $$\lambda=22.2\pm\sqrt{5.8^2+4.2^2}=22.2\pm 7.161 \;\Rightarrow\; \lambda_1=\boxed{29.36},\;\lambda_2=15.04.$$ Each pair sums to its trace ($10.26+1.24=11.5$; $29.36+15.04=44.4$), the arithmetic check.
4. Fraction of variance in PC1. Divide the leading eigenvalue by the trace: $$\text{PC1}_A=\frac{10.26}{11.5}=89.2\%,\qquad \text{PC1}_B=\frac{29.36}{44.4}=66.1\%.$$ For Group A a single component captures $\approx 89\%$ of the variance, so the two bands collapse to essentially one feature. For Group B, PC1 holds only $\approx 66\%$ and PC2 still carries a third of the information — both components are needed, so PCA achieves almost no reduction.

Explanation of the two points of view. Both groups applied the same, correct method; the difference lies entirely in their data. Group A imaged a scene whose two bands are strongly correlated ($r=0.78$): the data cloud is a long, thin ellipse (Figure 4, left), one eigenvalue dominates, and PC1 alone represents $89\%$ of the scene — so PCA successfully reduced two features to one with negligible loss, and Group A's enthusiasm is justified. Group B imaged a scene whose two bands are almost uncorrelated ($r=0.20$): the data cloud is nearly circular (Figure 4, right), the two eigenvalues are comparable ($29.4$ vs $15.0$), and PC1 captures only $66\%$, so neither component can be discarded without losing appreciable information. PCA gave Group B no useful dimensionality reduction — hence their view that it was of little value. The lesson: PCA's usefulness for feature reduction depends on the correlation (redundancy) between bands, not on the method itself.