Exploring The Empirical Privacy Landscape of Synthetic Data
AuthorsZihao Zhao, Avi Schwarzschild, Casey Meehan, Katherine Lee, Zhili Feng, Dylan Sam
AffiliationsJohns Hopkins University · OpenAI
The study finds that rewriting sensitive records to remove unique details can preserve model capabilities while reducing privacy leakage.
Key results
Synthetic Synthea records used for training.
Planted canary instances across five duplication levels.
Mean canary exposure in bits on Qwen3.5-4B-Base.
Mean canary exposure in bits on Qwen3.5-4B-Base.
Spearman correlation with measured verbatim leakage.
Spearman correlation with downstream capability.
What the paper found
This study compares privacy defenses for language models fine-tuned on synthetic medical records, using Synthea to create 5,000 training records and 2,340 planted-canary copies. It evaluates Qwen3.5-4B-Base and OpenAI’s gpt-oss-20b, contrasting direct fine-tuning and DP-SGD with rewriting by models including OpenAI’s GPT-5.5. The key finding is that uniqueness-aware privacy rewriting—which replaces rare, identifying details while preserving common clinical information—offers a better privacy-capability balance than ordinary paraphrasing: on Qwen3.5-4B-Base, mean canary exposure falls from 12.22 bits with direct fine-tuning to 2.21 bits with GPT-5.5 privacy-aware rewriting. DP-SGD also limits leakage, but at a substantial capability cost; direct fine-tuning retains the strongest capability while exposing the most information. More capable rewriting models generally improve privacy, though capability does not rise consistently with model size or added reasoning effort. An LLM’s privacy judgments track measured leakage, with a Spearman correlation of 0.89, but its quality judgments correlate negatively with downstream capability at -0.64. The results support privacy-aware rewriting as a promising empirical safeguard, not a formal differential-privacy guarantee.
Original abstract
Methods for training language models on sensitive data differ in how they balance capability gains against the risk of leaking private information. In this paper, we present a controlled comparison of data-level and training-level interventions for navigating the capability-privacy tradeoff on synthetic medical records with canaries to probe for leakage. Direct finetuning performs best on capability metrics but leads to the most privacy leakage, while DP-SGD reduces leakage at substantial cost to capabilities. Generating synthetic data from these records offers a more favorable balance, particularly when guided by a uniqueness-aware instruction to replace private details. Relative to naive paraphrasing, privacy-aware rewriting improves privacy protection across a variety of different rewriting models. We also find that more capable rewriting models generally reduce the risks of downstream privacy leakage, although held-out fit does not improve monotonically and increased test-time compute yields inconsistent gains. Overall, these results highlight privacy-aware rewriting as a promising direction for achieving a more favorable trade-off between capabilities and empirical privacy compared to other training-level methods, and our evaluation framework supports comparisons of future models and methods.
Read the original paperMore in AI Safety
Browse all 45 papers →Policy Is Not Capability: Some Safety Benchmarks Measure Which Definition of "Unsafe" You Trained On
Vahid Jebraeeli, Jonathon Romero, Yijin Wang, Adrian Chan, Jianbang Zhang
A guardrail’s safety score may say more about the policy used to label its training data than about its ability to detect harm.
SCOPE: Compiling Natural-Language Policies into Neurosymbolic Predicate Programs
Arnon Mazza, Elad Levi
SCOPE turns natural-language rules into interpretable programs that can enforce policies on LLM agents more reliably and cheaply.
RIDAR: Safeguarding Research Integrity in Autonomous Research Agents via Automated Detection and Runtime Intervention
Jiening Siow, Amy Xin, Junjie Wang, Ziyu Tang, Zijun Yao, Yunjia Qi, Fanjin Zhang, Lei Hou, Jie Tang, Juanzi Li
RIDAR detects research-integrity violations in autonomous research agents and intervenes during execution to reduce them.