Lethe: How Hard Is It to Forget? A Benchmark for Federated Unlearning in Medical Imaging

Lethe, from Shengchao Chen at the University of Technology Sydney and Ting Shu at Shenzhen University, is a benchmark for federated unlearning in medical imaging, evaluating 12 methods across 8 task families and 16 datasets, including MedMNIST, Kvasir-SEG, IXI MRI, and medical VQA, with client-, class-, and sample-level forgetting. Using retraining on retained data as the gold standard, the benchmark measures utility, privacy, backdoor erasure, efficiency, model closeness, and durability. Its central finding is that forgetting difficulty, rather than the unlearning algorithm, determines whether methods can be distinguished: ordinary client removal often produces nearly identical task performance because medical models generalize across hospitals, while hard class-level or sole-class removal exposes meaningful differences. On class-level forgetting, Prune and FedQUIT preserve retain accuracy close to retraining, whereas gradient ascent can collapse utility; even accuracy-matched methods may retain forget-class representations. Consequently, residual membership—not task accuracy—is often the key privacy signal: on BloodMNIST, FedRecovery reaches a calibrated membership-inference AUC of 0.572, compared with the ideal 0.5. Most methods are far cheaper than retraining; FedDNI achieves a 107.2x speedup on BloodMNIST, although FedEraser’s dependence on stored update history makes it slower than retraining in some settings. Lethe recommends difficulty-aware evaluation, membership-focused auditing, and deletion-ready architectures such as site-scoped adapters rather than relying solely on post-hoc weight editing.