OCR in the Foundation Model Era: From Specialist Models to Unified Models
AuthorsZhiheng Li, Zongyang Ma, Jiaxian Chen, Cheng Cui, Yi Liu, Ziqi Zhang, Chunfeng Yuan, Bing Li, Xiang Bai, Weiming Hu
A survey of how OCR is moving from specialized tools to vision-language models, and what those models still get wrong.
Key results
Overall score for hybrid regional parsing.
Overall score for specialized document VLM parsing.
After hard-example fine-tuning, up from 90.37.
Accuracy with pixel-reconstruction pretraining, up from 55.4%.
Score compared with GPT-4o at 736.
What the paper found
This survey reframes OCR as faithful transcription: every recovered word, formula, and structural relation must be supported by visible evidence in the image. It traces the shift from specialist pipelines to shared vision–language models and organizes current systems into five families: modular tools, document-specialized VLMs, hybrid workflows, general-purpose VLMs, and commercial services. Whole-page generation unifies content and structure, while hybrid methods use layout detection and region-level recognition to control cost and localize errors; both can still hallucinate, omit content, or lose spatial grounding. On OmniDocBench v1.6 full, the hybrid TeleOCR scores 96.91 overall and the specialized OvisOCR2 scores 96.47, showing that strong parsing does not require one workflow. Targeted training also matters: MinerU2.5-Pro raises table TEDS from 90.37 to 92.87 through fine-tuning on difficult examples, while MonkeyOCRv2 improves scrambled-text recognition from 55.4% to 72.1% with pixel-reconstruction pretraining. The survey reports OCRBench scores of 736 for GPT-4o and 931 for Qwen3.5, but stresses that diagnostic scores do not replace full-page evaluation. Its ecosystem includes OpenAI GPT-6, Google Gemini 3.1 Pro, and DeepSeek-OCR-2; the central conclusion is that unified models, specialists, and hybrids will coexist, and evaluation must better test multilingual, multipage, and real-camera documents.
Original abstract
Optical Character Recognition (OCR) is among the oldest and most widely deployed applications of artificial intelligence. This survey reviews its development, focusing on the Foundation Model Era. We define OCR through faithful transcription: within a specified visual scope, OCR should completely reconstruct the text, notation, and document structure observable in the source image, and every textual, notational, or structural element in the output must be verifiable from visual evidence in that image. We trace the development from specialist OCR models toward unified models, examining joint text detection and recognition, structured sequence generation, and the use of pretrained vision-language models (VLMs). We organize contemporary OCR systems into five coexisting families: modular toolkits and task-specific models, specialized document VLMs, hybrid workflows, general-purpose VLMs, and commercial products and managed services. We compare their architectures, data, and training, examining hallucination and omission, insufficient spatial grounding, and inference cost in generative OCR. We review evaluation from character and scene text recognition to parsing individual structural elements and full-page transcription, highlighting gaps in consistent output definitions, attribution of performance to individual components, and coverage of real capture conditions and visible content and structure. Finally, we summarize the challenges facing OCR and discuss directions for future research. We continuously track related work at https://github.com/zhihengli-casia/Awesome-OCR-in-the-Foundation-Model-Era.
Read the original paperMore in Multimodal AI
Browse all 79 papers →ALIVE: Making Streaming Video Models Interactive
Jinzhao Li, Yuan Xu, Jinxuan Wu, Yukun Wang, Yijia Lei, Hanyu Ji, Yinuo Chen, Zining Huang, Chi Zhang, Hang Ye, Yiwu Zhong, Honglei Yan, Panwang Pan, Yizhou Wang, Yin Li, Miao Liu
ALIVE aims to make video-streaming AI an interactive partner that can monitor events, coordinate tasks, and ask for feedback.
Let Samples Sing in Chorus: Mitigating Spurious Correlations via Cross-view Consensus
Weiwei Li, Junzhuo Liu, Tong Chu, Hengfu Yu, Yinjie Lei, Wen Li
CHORD combines clues from multiple representation spaces to help models rely less on misleading shortcuts and perform better on underrepresented groups.
LightTAR: Light and Fast Target-Aware Representations for Multimodal Tabular Learning
Eilam Shapira, Alan Arazi, Shoham Grunblat, Roi Reichart
LightTAR adds a small set of target-aware features to standard embeddings, helping tabular models make better predictions with little extra runtime.