NTH

The uneven rise of AI-generated web content

AuthorsAdam Visokay, Soumyakanti Pan, Molly Griston, Polina Iasakova, Mark Nepf, Denis Peskoff, Riya Sinha, Jikai Sun, Isabelle Thiara, Letao Wang, Yiwei Wu, Lauren Yan, Tyler H. McCormick

AffiliationsSchool of Information, University of California, Berkeley · Department of Statistics, University of Washington · Department of Physics, University of Colorado, Boulder · Department of Statistics, Oregon State University · Evans School of Public Policy, University of Washington · Department of Information, University of Texas at Austin · Hubbard School of Journalism and Mass Communication, University of Minnesota · Department of Sociology, Washington University in St. Louis · Department of Family Medicine & Community Health, University of Kansas Medical Center · Department of Sociology, University of Washington

November 12, 2026 2 min read
Watch on YouTube
The one-line take

A study of over five million webpages finds that AI-written text is spreading unevenly, with the fastest growth on less-central pages.

Key results

8.1
Annual prevalence increase

Percentage points per year from January 2025 to August 2026.

78.8%
Peripheral tiers' share of prevalence increase

Combined contribution from the two least-central PageRank tiers.

32.8%
English-only page prevalence

Estimated share in August 2026.

29.5%
.com page prevalence

Estimated share in August 2026.

What the paper found

Across 71 Common Crawl snapshots, the study analyzed 5.1M webpages and estimates that 23.6% contained fully AI-generated or AI-edited text by August 2026. Prevalence is rising faster: the annual increase reached 8.1 percentage points after January 2025. The spread is uneven: the two least-connected PageRank tiers accounted for 78.8% of the overall increase, while English-only pages reached 32.8% prevalence, compared with 17.0% for pages with no English detected. By domain, prevalence was highest on .com pages at 29.5%, and much lower on .gov and .edu pages. To address detector errors that vary by language, text length, and domain, the researchers combine Open Pangram, a classifier fine-tuned on Llama-3.2-3B, with Binoculars using Falcon-7B, then estimate prevalence in a Bayesian hierarchical model calibrated against labeled benchmarks and pre-2022 crawls. Those benchmarks include text generated by OpenAI’s GPT-4.1, Anthropic’s Claude Sonnet 4, Google’s Gemini 2.5 Flash, and Llama-3.3-70B. The results show why raw detector flags can mislead: the model corrects for false positives and missed detections, carries that uncertainty into its estimates, and adjusts results to represent the wider crawlable web.

Original abstract

In less than five years, generative artificial intelligence (AI) has dramatically reshaped the public-facing web. Using a sample of more than 5 million crawlable webpages, we demonstrate that the share containing AI-generated text has increased precipitously, but unevenly. Our analyses reveal substantial differences in AI composition across language and domain categories (e.g., .edu, .com, or .gov). A higher fraction of English-language webpages contain AI-generated text compared to pages in other languages, and prevalence is substantially lower on .gov and .edu domains compared to .com. These distinctions persist even after accounting for compositional effects. We also find heterogeneity based on a page's position in the connectivity structure of the web. AI-generated text is most common and has increased most rapidly in pages that are the least central in the graph, which drives much of the overall growth in AI-generated text. A slower increase has taken place for more central pages, with the most central tier emerging as the least likely to contain AI-generated text. Methodologically, we use multiple AI detection tools, and generate estimates using a Bayesian hierarchical model that incorporates corrections for the detectors' statistical properties (both false positives and false negatives) and allows for different detector performance across groups, while propagating uncertainty in detector error rates into prevalence estimates. Our work suggests that AI is a prominent, but varied, contributor to public information and internet-based communication.

Read the original paper

More in Natural Language Processing

Browse all 29 papers →
01Nlp

Contextual Embeddings Beyond the Gold Chunk

Sedigheh Eslami, Markus Krimmel, William Barber, Antoine Chaffin, Maximilian Schall, Jonas Golde, Louis Milliken, Ekaterina Glazkova, Denis Bykov, Bo Wang

A new retriever learns to surface both the chunks that answer a question and the evidence that supports those answers.

Read analysis