Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloning
AuthorsYoomee Cho, Jisun Lee
AffiliationsESTsoft, Seoul, Republic of Korea
A training-free technique helps voice-cloning systems preserve who is speaking while making their accent sound native to the target language.
Key results
Accent score out of 5 for held-out Korean-to-English dubbing with AAG.
Speaker-similarity advantage over guidance reweighting at equal accent.
OmniVoice speaker-similarity advantage at equal accent.
Seven tested variants stayed less than this amount above the usual trade-off curve.
Decided responses favoring AAG over equal-accent reweighting for reference-speaker closeness.
What the paper found
Cross-lingual voice cloning can preserve a speaker’s identity but carry their source-language accent into the target language. Accent analogy guidance (AAG) is a training-free sampler technique that estimates the accent direction by comparing a model’s predictions for the same synthetic proxy voice speaking two languages, then subtracts that direction while keeping the speaker constant. Across OmniVoice, MaskGCT, F5-TTS, and CosyVoice 2, AAG improves accent at a given speaker similarity, while seven inference-time alternatives remain less than 0.05 above the models’ usual identity–accent trade-off curve. On held-out Korean-to-English dubbing, OmniVoice reaches an accent score of 4.28 out of 5, close to 4.33 for speech generated without a speaker reference; its speaker similarity is 0.294, versus 0.02 retained by ordinary guidance reweighting at the same accent, giving a ΔSIM of 0.27. On Zeroth-Korean, AAG’s ΔSIM is 0.11. The accent ratings use a blind Gemini 2.5 Pro judge, with Whisper large-v3 language identification and a 12-listener panel providing additional checks; 85% of decided listener responses favored AAG over equal-accent reweighting for closeness to the reference speaker. A premise test also predicts the exception: X-Voice shows little accent dependence on the reference language, so AAG has no useful accent direction to remove.
Original abstract
In cross-lingual zero-shot text-to-speech, the accent of the reference leaks into the target speech. We propose accent analogy guidance (AAG), a training-free sampler term that subtracts an accent direction estimated from the model's own predictions for one synthetic voice rendered in both languages, so the voice cancels and only the accent remains. By a blind LLM accent judge on real dubbing data, reweighting classifier-free guidance between reference and text, and its variants, stay near one identity-accent trade-off curve; we score a method by its speaker similarity above that curve at equal accent ($Δ$SIM). Across four open TTS models AAG lies above the curve: on OmniVoice $Δ$SIM is +0.11 to +0.27 on three test sets (accent 3.51 to 4.28 on a 1-5 scale at speaker similarity 0.29, where reweighting keeps 0.02); MaskGCT and CosyVoice 2 also lie above their curves, and on F5-TTS it is more native than any reweighting setting. An LLM-free language-ID measure and a twelve-listener panel agree. A premise test and the reach of a model's own curve indicate in advance whether and roughly how much AAG can gain, predicting the one model where it gains nothing (X-Voice).
Read the original paperMore in Speech AI
Browse all 30 papers →DEFINE: Exemplar-Guided Accent Control for Zero-Shot TTS
Ambuj Mehrish, Abhinaba Roy, Alex Ivanov, Tawsif Ahmed, Dorien Herremans
DEFINE lets a zero-shot speech synthesizer preserve one speaker’s identity while dialing in an accent from a separate audio example.
HiPLEX: Hierarchical Policy Factorization for Full Duplex Speech Language Models
Kyudan Jung, Hyunsin Park, Yoonhyung Lee, Jinhwan Park, Jinhyeok Yang, KiHyun Nam, Jaegul Choo, Jinkyu Lee
HiPLEX teaches speech AI to better manage conversations by learning separately when to speak and what to say.
Pruned CTC for Memory-Efficient Large-Vocabulary ASR Training
Yifan Yang, Xiaoyu Yang, Zengrui Jin, Xian Shi, Yuxuan Wang, Yu Xi, Ziyang Ma, Qi Chen, Ruiyang Xu, Hui Wang, Dongchao Yang, Jin Xu, Xie Chen
A new CTC training strategy makes large-vocabulary LLM speech recognition far more memory-efficient while retaining competitive accuracy and fast streaming inference.