TF-IDF and BM25 Are Exact KL Divergences

This paper gives TF-IDF and BM25 a unified probabilistic foundation by proving that their term-relevance scores arise from Kullback-Leibler divergences between explicitly constructed probability models. The derivation uses 4 random variables: document index, token, key-token status, and common-token status. For TF-IDF, the divergence assigns token probability d_i(t)/|d_i| and corpus rarity probability DF(t)/N, yielding the standard product of normalized term frequency and inverse document frequency exactly. For the practical BM25 variant used in production search software such as Milvus, the paper replaces these probabilities with a surrogate model incorporating term-frequency saturation through k1 and document-length normalization through b; the resulting divergence equals BM25 after the known scalar rescaling 1/(k1+1). Its IDF term includes the standard +1 correction, while the original BM25 formula without that correction is represented as a difference between divergences involving an adversarial probability model. In both cases, a query score is the query-token-weighted average of term divergences, measuring how strongly uncommon query keywords in a document depart from a naive commonness assumption. The result is theoretical rather than empirical: it supplies an exact information-theoretic interpretation and makes TF-IDF and BM25 directly comparable with other divergence-based retrieval methods.