Distill Once, Adapt Life Long Exploring Dataset Distillation for Continual Test Time Adaptation

DO-ALL, or Distill Once, Adapt Life-Long, reframes continual test-time adaptation by replacing raw source replay with a compact distilled anchor bank built once before deployment through dataset distillation. Each anchor stores a synthetic image, a soft source-model label, and a latent feature vector, and at test time each incoming target sample is matched to its nearest anchor in cosine feature space so adaptation stays semantically grounded instead of drifting under long non-stationary streams. The method then combines anchor-based replay, target-anchor MixUp regularization, layer-wise MMD alignment, and harm-adaptive parameter blending to suppress catastrophic forgetting while preserving useful online updates. On ImageNet-C at corruption severity 5, DO-ALL reduces error from 58.0% to 56.6% for EATA, from 59.8% to 57.4% for RMT, and from 54.5% to 53.6% for ROID; on the CCC benchmark, ROID improves from 33.6% to 34.7%, with the hard split rising from 13.2% to 15.5%. The ablations show that the full combination beats the baseline by 0.9 points, and even a minimal anchor bank with IPC=1 already helps, while larger anchor budgets further strengthen stability. Across distillation methods such as WMDD, DELT, and SRe2L, and even across backbones including ResNet-18, ResNet-50, and ViT-B, the distilled anchors remain effective, indicating that DO-ALL acts as a plug-and-play stabilization layer for privacy-constrained, long-horizon CTTA.