Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing

This paper from nyra labs argues that verbatim versus intended transcription is an uncontrolled latent variable in modern ASR, destabilizing decoding, contaminating WER, and making word timing ambiguous. Starting from OpenAI’s Whisper-medium, the authors add coverage-aware decoder mode tags that explicitly select verbatim or intended output, supervised cross-attention alignment heads for timing, and a transcript-conditioned “verbatimize” mode that restores acoustically grounded disfluencies to an intended transcript. Training only newly added tag embeddings, with Whisper’s pretrained encoder and decoder frozen, raises German disfluency event F1 to 79%; full English-only fine-tuning transfers zero-shot to German at 93.8% event F1. On FluencyBank, supervised attention produces 102 ms mean absolute word-boundary error, outperforming forced alignment at 142 ms and WhisperX at 200 ms. Verbatimize increases rare-word recall from 6.8% to 96.1% on the ICSI rare-word set, while casing perturbation helps the model copy exact transcript content. The study also finds that up to 60% of reported WER on conversational benchmarks can reflect style mismatch rather than recognition errors. Comparisons include NVIDIA’s Canary-1B and WhisperX; GPT-4o generated intended transcript variants and Claude by Anthropic assisted some coding. The central conclusion is that pretrained ASR models often possess verbatim capability already, but explicit policy controls are needed to activate it reliably across languages and downstream timing tasks.