The Embedder's Dilemma: LLMs Are Better, but at What Cost?

The paper introduces MTEB(LLM), a controlled cost-aware comparison of 10 LLMs and 26 embedding models across 37 tasks covering classification, semantic textual similarity, clustering, pair classification, and retrieval. Google’s Gemini 3.1 Pro scored 77.6 overall, while the embedding model Octen-8B reached 77.2, leaving only a 0.4-point difference. The task breakdown is more important than the aggregate: embedding models led classification by 5.6 points, with SFR-2 scoring 90.8 versus Gemini’s 85.2, while LLMs led reasoning-heavy retrieval by 8.5 points, scoring 64.5 versus Octen-8B’s 56.0. Clustering, STS, and pair classification were effectively tied. That parity is expensive: one Gemini 3.1 Pro benchmark pass cost $154 versus $0.11 for Octen-8B, a 1431-times gap. On the same NVIDIA H100, open LLMs processed 2.5 to 736 times fewer tokens per second than embedding models. Reasoning tokens consumed 28 to 81 percent of LLM inference cost, yet reducing reasoning tokens by 54 to 96 percent preserved or improved retrieval for most models. On BRIGHT, an LLM reranker improved Qwen3-E-8B retrieval from 22.3 to 35.1 nDCG@10, whereas on semantic BEIR, the embedding model alone scored 63.1, beating reranked variants. The recommendation is a hybrid pipeline: embeddings for high-throughput similarity, classification, and clustering, with LLMs such as Gemini, DeepSeek-R1, or Qwen reserved for reasoning-intensive retrieval; OpenAI’s GPT-5 and Anthropic’s Claude Opus 4.6 were discussed but not evaluated.