Your Embedding Model is SMARTer Than You Think

Your Embedding Model is SMARTer Than You Think, from UW-Madison, Korea University, and NetApp, introduces SMART, a single-to-multi adaptation for retrieval transformers that turns off-the-shelf single-vector multimodal embedders into late-interaction retrievers without retraining from scratch. The core insight is that contrastive supervision on a pooled <eot> embedding already shapes the geometry of the model’s remaining hidden states, so SMART reuses those frozen states with ColBERT-style MAXSIM and combines them with the original pooled cosine score in a simple hybrid objective. On a controlled 5×5 code–marker binding benchmark with 1000 queries, the pooled single-vector score reaches only 31.9% pairwise accuracy, while late interaction over final-layer hidden states reaches 56.8%. On MMEB-V2, inference-only SMART improves the Qwen3-VL-Embedding-8B average from 78.83% to 79.34%, and the 2B model from 74.87% to 75.77%, while also lifting VLM2Vec-V2.0 from 64.50% to 67.04%. With a lightweight frozen-backbone adapter trained for 1 hour and 50 minutes on eight A6000 GPUs, Qwen3-VL-Embedding-2B surpasses jina-embeddings-v4 by 0.34 points on the visdoc subset. In a LoRA conversion study, a SMART-based single-vector-to-multi-vector conversion trained in 9.5 hours trails a from-scratch multi-vector model by only 0.63 average points, showing that SMART can recover local evidence, improve retrieval, and cut training compute by about 20%.