Contextual Embeddings Beyond the Gold Chunk
AuthorsSedigheh Eslami, Markus Krimmel, William Barber, Antoine Chaffin, Maximilian Schall, Jonas Golde, Louis Milliken, Ekaterina Glazkova, Denis Bykov, Bo Wang
AffiliationsPerplexity AI · Equal contribution, †Work done while at Perplexity · turbopuffer
A new retriever learns to surface both the chunks that answer a question and the evidence that supports those answers.
Key results
Base-query count in the benchmark.
Score for pplx-embed-v2-context-9b.
Score for pplx-embed-v2-context-9b.
Score for pplx-embed-v2-context-9b across eight tasks.
The 9B model uses four times less index storage than Nemotron-3.5-8B in the stated comparison.
What the paper found
This paper introduces Perplexity’s pplx-embed-v2-context, a 9B and 0.6B model family that learns which document chunks answer a query and which provide evidence to interpret or verify that answer. Instead of treating every non-gold chunk as a negative, the models distill graded, token-level relevance scores from a query-aware context compression model into chunk targets, then train with a chunk-level KL-divergence loss and document-level InfoNCE. This makes supervision independent of chunk boundaries and avoids adding a teacher or reranking step at retrieval time. The researchers also introduce context-bench and Q2D-Chunk, which evaluate evidence retrieval alongside answer retrieval; Q2D-Chunk contains 6069 base queries. On Q2D-Chunk, the 9B model reaches 86.4% Answer@10, compared with 83.9% for Voyage-context-4. It scores 45.5% Answer@10 on context-bench, 14.4 percentage points above Voyage-context-4, and achieves 83.4% average nDCG@10 on ConTEB. Beyond text, the model transfers visual retrieval abilities from its vision-language backbone: splitting document images into horizontal chunks improves results on ViDoRe-V3, and the 9B model with two chunks and 1024-dimensional embeddings outperforms NVIDIA Nemotron-3.5-8B while using an index four times smaller.
Original abstract
Contextual embeddings represent chunks of a document with respect to the document's global context. These models are typically trained and evaluated under the assumption that a relevant document contains only a single relevant chunk. In this work, we argue that in real-world use cases a retriever should not only find answers spread over several chunks but also surface the evidence needed to verify them. To address this, we introduce pplx-embed-v2-context, a family of contextual embedding models trained by distilling graded chunk-level relevance from a query-aware context compression model. This removes the need for gold-chunk labels and makes the training targets independent of the chunking. We utilize a vision-language backbone for training and release pplx-embed-v2-context in 9B and 0.6B sizes with compatible embedding spaces. On the evaluation side, we introduce two benchmarks that label both the chunks that answer a query and the evidence needed to verify the answer. Context-bench is a controlled diagnostic, and Q2D-Chunk is built from real-world search queries. To encourage further research, we also publish Q2D-Chunk as a public leaderboard. Our 9B model sets a new state of the art on both benchmarks and ConTEB. We find that the visual retrieval capabilities of our 9B model transfer from its vision-language base model and propose to extend chunking to image inputs. We demonstrate that our models achieve state-of-the-art performance among dense retrievers on visual ViDoRe-V3 and that performance improves with granular image chunking.
Read the original paperMore in Natural Language Processing
Browse all 29 papers →Grammar Control for Language Learning with Sequential Monte Carlo and Small Language Models
Kate McCurdy, Jacob Hoover Vigly, Thomas Hikaru Clark, Karolina Stańczak, Yahya Emara, Joshua B. Tenenbaum, Alexander K. Lew, Ryan Cotterell, Timothy J. O'Donnell
A sampling-based method helps small language models produce dialogue that reliably practices specific grammar skills.
The uneven rise of AI-generated web content
Adam Visokay, Soumyakanti Pan, Molly Griston, Polina Iasakova, Mark Nepf, Denis Peskoff, Riya Sinha, Jikai Sun, Isabelle Thiara, Letao Wang, Yiwei Wu, Lauren Yan, Tyler H. McCormick
A study of over five million webpages finds that AI-written text is spreading unevenly, with the fastest growth on less-central pages.
AdaTutoRank: Learning to Rerank Document Sets via Adaptive Tutoring Optimization for RAG and Deep Research
Kailin Jiang, Lei Liu, Jian Xi, Yangqi Chen, Hui Xu, Hongwei Zhao, Bin Li, Yu Lu, Haibo Shi
AdaTutoRank teaches rerankers to assemble complementary evidence sets rather than merely picking individually relevant documents, improving RAG and deep-research retrieval with fewer calls.