NTH

Contextual Embeddings Beyond the Gold Chunk

AuthorsSedigheh Eslami, Markus Krimmel, William Barber, Antoine Chaffin, Maximilian Schall, Jonas Golde, Louis Milliken, Ekaterina Glazkova, Denis Bykov, Bo Wang

AffiliationsPerplexity AI · Equal contribution, †Work done while at Perplexity · turbopuffer

November 12, 2026 2 min read
Watch on YouTube
The one-line take

A new retriever learns to surface both the chunks that answer a question and the evidence that supports those answers.

Key results

6069
Q2D-Chunk base queries

Base-query count in the benchmark.

86.4%
Q2D-Chunk Answer@10

Score for pplx-embed-v2-context-9b.

45.5%
Context-bench Answer@10

Score for pplx-embed-v2-context-9b.

83.4%
ConTEB average nDCG@10

Score for pplx-embed-v2-context-9b across eight tasks.

4
Visual retrieval index reduction

The 9B model uses four times less index storage than Nemotron-3.5-8B in the stated comparison.

What the paper found

This paper introduces Perplexity’s pplx-embed-v2-context, a 9B and 0.6B model family that learns which document chunks answer a query and which provide evidence to interpret or verify that answer. Instead of treating every non-gold chunk as a negative, the models distill graded, token-level relevance scores from a query-aware context compression model into chunk targets, then train with a chunk-level KL-divergence loss and document-level InfoNCE. This makes supervision independent of chunk boundaries and avoids adding a teacher or reranking step at retrieval time. The researchers also introduce context-bench and Q2D-Chunk, which evaluate evidence retrieval alongside answer retrieval; Q2D-Chunk contains 6069 base queries. On Q2D-Chunk, the 9B model reaches 86.4% Answer@10, compared with 83.9% for Voyage-context-4. It scores 45.5% Answer@10 on context-bench, 14.4 percentage points above Voyage-context-4, and achieves 83.4% average nDCG@10 on ConTEB. Beyond text, the model transfers visual retrieval abilities from its vision-language backbone: splitting document images into horizontal chunks improves results on ViDoRe-V3, and the 9B model with two chunks and 1024-dimensional embeddings outperforms NVIDIA Nemotron-3.5-8B while using an index four times smaller.

Original abstract

Contextual embeddings represent chunks of a document with respect to the document's global context. These models are typically trained and evaluated under the assumption that a relevant document contains only a single relevant chunk. In this work, we argue that in real-world use cases a retriever should not only find answers spread over several chunks but also surface the evidence needed to verify them. To address this, we introduce pplx-embed-v2-context, a family of contextual embedding models trained by distilling graded chunk-level relevance from a query-aware context compression model. This removes the need for gold-chunk labels and makes the training targets independent of the chunking. We utilize a vision-language backbone for training and release pplx-embed-v2-context in 9B and 0.6B sizes with compatible embedding spaces. On the evaluation side, we introduce two benchmarks that label both the chunks that answer a query and the evidence needed to verify the answer. Context-bench is a controlled diagnostic, and Q2D-Chunk is built from real-world search queries. To encourage further research, we also publish Q2D-Chunk as a public leaderboard. Our 9B model sets a new state of the art on both benchmarks and ConTEB. We find that the visual retrieval capabilities of our 9B model transfer from its vision-language base model and propose to extend chunking to image inputs. We demonstrate that our models achieve state-of-the-art performance among dense retrievers on visual ViDoRe-V3 and that performance improves with granular image chunking.

Read the original paper

More in Natural Language Processing

Browse all 29 papers →
02Nlp

The uneven rise of AI-generated web content

Adam Visokay, Soumyakanti Pan, Molly Griston, Polina Iasakova, Mark Nepf, Denis Peskoff, Riya Sinha, Jikai Sun, Isabelle Thiara, Letao Wang, Yiwei Wu, Lauren Yan, Tyler H. McCormick

A study of over five million webpages finds that AI-written text is spreading unevenly, with the fastest growth on less-central pages.

Read analysis