Projection Pursuit CPCANet for Domain Generalization
AuthorsYu-Hsi Chen, Abd-Krim Seghouane
Resources
PP-CPCANet replaces fragile covariance estimation with a robust, jointly learned orthogonal basis to improve domain-generalized representations.
Key results
Average accuracy across the four domain-generalization benchmarks with an SSM-based backbone.
Baseline average accuracy for comparison with PP-CPCANet-B.
Peak GPU memory in GB for PP-CPCANet with ResNet-50.
Best initial projection dimension in the ablation study.
Single-depth cascade produced the best ablation accuracy.
Average accuracy obtained with projection dimension 128 and cascade depth 1.
What the paper found
Yu-Hsi Chen and Abd-Krim Seghouane at the University of Melbourne propose Projection Pursuit CPCANet, or PP-CPCANet, for domain generalization when mini-batch covariance estimates become rank-deficient. Instead of computing batch-wise Common Principal Component Analysis, the method learns a global orthogonal basis on the Stiefel manifold through a Cayley transform. Its projection-pursuit objective combines symmetry-breaking component weights with a detached-median L1 dispersion score, producing dense, robust gradients despite outliers and small sample sizes. PP-CPCANet also retains domain-guided feature modulation and a progressive bottleneck design, although ablations show that a single cascade depth is best. Across PACS, VLCS, OfficeHome, and TerraIncognita, the method achieves competitive or state-of-the-art domain-generalization accuracy with ResNet-50, DeiT, and VMamba backbones. With the SSM-based PP-CPCANet-B configuration, it reaches 77.2 percent average accuracy, compared with 76.6 percent for CPCANet-B, while the ResNet-50 version uses 8.25 GB of peak GPU memory versus CPCANet’s 8.65 GB. The strongest ablation uses an initial projection dimension of 128 and cascade depth 1, achieving 69.2 percent average accuracy, supporting the paper’s conclusion that covariance-free projection pursuit can improve optimization stability without requiring deeper cascades.
Original abstract
Domain Generalization (DG) aims to learn representations robust to distribution shifts. Recent geometric alignment methods, such as CPCANet, extract domain-invariant structures through batch-wise Common Principal Component Analysis (CPCA). However, CPCANet suffers from rank-deficient covariance estimation due to the small-sample-size issue in mini-batch training. To address this limitation, we propose Projection Pursuit CPCANet (PP-CPCANet), a covariance-free framework that learns a global orthogonal basis on the Stiefel manifold and jointly optimizes it with network parameters via the Cayley transform. We further introduce a symmetry-breaking detached-median PP dispersion objective to extract common principal components (CPCs) with dense and robust optimization signals. Experiments on four DG benchmarks show that PP-CPCANet achieves SOTA performance while maintaining stable training.
Read the original paperMore in Neural Networks
Browse all 22 papers →End-to-End Hard-Label Cryptanalytic Model Extraction Using Efficient Sign Recovery
Akira Ito, Takayuki Miura, Yosuke Todo
A new query-efficient technique makes it possible to steal the parameters of small black-box neural networks using only their predicted labels.
Retrieving Individual Stems from Music Mixtures with Slot Embeddings
David Braun, Junyi Fan, Pranay Manocha, Donald S. Williamson, Adam Finkelstein
Stembed lets music producers search for individual instrument sounds hidden inside a full song by representing the mixture as multiple searchable stem-like embeddings.
The Linear Representation Hypothesis Needs a Group Action
Louie Hong Yao, Yuhao Li, Shengchao Liu
This paper argues that claims about linear representations only become meaningful once we specify which transformations leave a representation essentially unchanged.