NTH

OCR in the Foundation Model Era: From Specialist Models to Unified Models

AuthorsZhiheng Li, Zongyang Ma, Jiaxian Chen, Cheng Cui, Yi Liu, Ziqi Zhang, Chunfeng Yuan, Bing Li, Xiang Bai, Weiming Hu

November 14, 2026 2 min read
Watch on YouTube
The one-line take

A survey of how OCR is moving from specialized tools to vision-language models, and what those models still get wrong.

Key results

96.91
TeleOCR OmniDocBench v1.6 Overall

Overall score for hybrid regional parsing.

96.47
OvisOCR2 OmniDocBench v1.6 Overall

Overall score for specialized document VLM parsing.

92.87
MinerU2.5-Pro table TEDS

After hard-example fine-tuning, up from 90.37.

72.1%
MonkeyOCRv2 scrambled-text recognition

Accuracy with pixel-reconstruction pretraining, up from 55.4%.

931
Qwen3.5 OCRBench

Score compared with GPT-4o at 736.

What the paper found

This survey reframes OCR as faithful transcription: every recovered word, formula, and structural relation must be supported by visible evidence in the image. It traces the shift from specialist pipelines to shared vision–language models and organizes current systems into five families: modular tools, document-specialized VLMs, hybrid workflows, general-purpose VLMs, and commercial services. Whole-page generation unifies content and structure, while hybrid methods use layout detection and region-level recognition to control cost and localize errors; both can still hallucinate, omit content, or lose spatial grounding. On OmniDocBench v1.6 full, the hybrid TeleOCR scores 96.91 overall and the specialized OvisOCR2 scores 96.47, showing that strong parsing does not require one workflow. Targeted training also matters: MinerU2.5-Pro raises table TEDS from 90.37 to 92.87 through fine-tuning on difficult examples, while MonkeyOCRv2 improves scrambled-text recognition from 55.4% to 72.1% with pixel-reconstruction pretraining. The survey reports OCRBench scores of 736 for GPT-4o and 931 for Qwen3.5, but stresses that diagnostic scores do not replace full-page evaluation. Its ecosystem includes OpenAI GPT-6, Google Gemini 3.1 Pro, and DeepSeek-OCR-2; the central conclusion is that unified models, specialists, and hybrids will coexist, and evaluation must better test multilingual, multipage, and real-camera documents.

Original abstract

Optical Character Recognition (OCR) is among the oldest and most widely deployed applications of artificial intelligence. This survey reviews its development, focusing on the Foundation Model Era. We define OCR through faithful transcription: within a specified visual scope, OCR should completely reconstruct the text, notation, and document structure observable in the source image, and every textual, notational, or structural element in the output must be verifiable from visual evidence in that image. We trace the development from specialist OCR models toward unified models, examining joint text detection and recognition, structured sequence generation, and the use of pretrained vision-language models (VLMs). We organize contemporary OCR systems into five coexisting families: modular toolkits and task-specific models, specialized document VLMs, hybrid workflows, general-purpose VLMs, and commercial products and managed services. We compare their architectures, data, and training, examining hallucination and omission, insufficient spatial grounding, and inference cost in generative OCR. We review evaluation from character and scene text recognition to parsing individual structural elements and full-page transcription, highlighting gaps in consistent output definitions, attribution of performance to individual components, and coverage of real capture conditions and visible content and structure. Finally, we summarize the challenges facing OCR and discuss directions for future research. We continuously track related work at https://github.com/zhihengli-casia/Awesome-OCR-in-the-Foundation-Model-Era.

Read the original paper

More in Multimodal AI

Browse all 79 papers →
01Multimodal

ALIVE: Making Streaming Video Models Interactive

Jinzhao Li, Yuan Xu, Jinxuan Wu, Yukun Wang, Yijia Lei, Hanyu Ji, Yinuo Chen, Zining Huang, Chi Zhang, Hang Ye, Yiwu Zhong, Honglei Yan, Panwang Pan, Yizhou Wang, Yin Li, Miao Liu

ALIVE aims to make video-streaming AI an interactive partner that can monitor events, coordinate tasks, and ask for feedback.

Read analysis