How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning

Kaizhen Tan, Heqing Du, and Yang Feng of Carnegie Mellon University and Columbia University introduce a compression-based instrument for measuring how many information bits a parameter-efficient adapter actually writes into a frozen model. Using 64-token uniformly random sequences over a 2,048-symbol vocabulary, yielding 704 bits per sequence, they separate memorization from prediction and show that LoRA adapters store substantially less than full fine-tuning: full models reach 3.6 bits per parameter, while adapters plateau between 1.7 and 2.8 bits per trainable parameter under a matched 16,000-step budget. Capacity is not determined by parameter count alone: with about 37K trainable parameters, moving the update from attention to the MLP changes capacity from 1.30 to 2.43 bits per parameter, while reduced-precision storage has little effect. The frozen base also matters dramatically: the same adapter memorizes 29% of random data on a randomly initialized base, versus 98% on a WikiText-pretrained base, while full fine-tuning of the random base reaches 99%. On Qwen2.5-0.5B, canary exposure tracks written bits rather than nominal rank; increasing rank from 16 to 64 raises exposure by 11 bits. Finally, comparing supervised fine-tuning with GRPO on verifiable rewards at similar task performance, supervised training continues writing training-specific information and copies injected secrets, whereas GRPO’s excess behavior-write signal remains indistinguishable from zero. The results frame LoRA adapters, including those shared through model hubs or federated systems, as measurable data-bearing artifacts rather than harmless skill patches.