Contrastive Distribution Matching for Amortized Sequential Monte Carlo in Discrete Diffusion

Contrastive Distribution Matching, or CDM, addresses a core bottleneck in discrete diffusion reward alignment: Twisted Sequential Monte Carlo needs an optimal twist function, but in discrete state spaces that twist is usually estimated by expensive Monte Carlo over clean samples, which makes inference slow, especially when the reward is costly to evaluate. The paper’s key idea is to amortize that cost by learning the twist with a contrastive forward-KL objective, so the learned twist can be evaluated in a single lightweight head attached to the diffusion backbone, adding less than 5% runtime overhead, as low as 0.5% in some setups. CDM replaces regression-style Soft Value training with positive and negative samples: positive samples come from the reward-tilted target, negative samples from the current approximation, and the gradient explicitly upweights high-reward regions while suppressing poor ones. A diffusion-native trick reuses clean positive samples through the closed-form forward kernel, enabling multiple updates per sample and reducing reward calls. Across toxic text generation on OpenWebText, regulatory DNA sequence design on a 700,000-sequence enhancer dataset, protein designability with DPLM-2 and ESMFold-based self-consistency, and diffusion LLM alignment with LLaDA-8B-Instruct on RewardBench, CDM consistently beats Best-of-N, SMC with M=1 or 4, and Soft Value under matched wall-clock time, while remaining compatible with fine-tuned proposals such as d1 and DRAKES and mitigating their mode collapse.