Skip to main content
How AI agents discover and consume your documentation. Agents can read documentation directly on behalf of users. Agent traffic now rivals human traffic and is on pace to become the majority of all traffic to documentation sites. The discovering layer is about the machines that consume your content based on humans’ prompts and queries. Data comes from across 17,788 active documentation sites.

​
Seven day traffic

​
Twenty four hour traffic

46% of all active documentation sites were touched by suspected scrapers in a single day.

​
The scraper fingerprint

Scrapers are identified by behavioral patterns. High event volume from a user agent and IP pair that otherwise looks like a normal browser:

​
Who’s reading: named AI agents

These are AI clients that explicitly identify themselves in their HTTP user agent string. Named agents are a small fraction of total AI reads, the majority of which arrive via unidentified scrapers. Claude leads by a factor of 15 times over the next closest named agent (Bytedance at 38K).
“Claude” here is a group of distinct user agents: ClaudeBot (Anthropic’s crawler), Claude-User (real-time responses to user queries), and Claude Code (the coding agent fetching docs programmatically). They all show up in the user agent string and we group them.

​
Where the traffic comes from: cloud providers

Most traffic arrives via Cloudflare edge nodes, so the recorded IP is usually a CDN proxy—not the originating client. “Other” has the highest IP diversity (92K unique IPs), largely residential. The cloud provider table reflects where requests exit the network, not where they originate.

​
AI-native endpoints

These are the endpoints that are classified as high certainty AI reads: These endpoints were purpose-built for machine consumption.

​
Benchmark

AI readership: 35% This is a conservative estimate of agent readership based on the higher confidence signals of agent views.

​
What this means for the craft

You are no longer writing for just humans.
  • Structure matters as much as ever. Agents need well structured content to parse and process. People do too.
  • llms.txt is the entry point for agent discovery. Agents need to know where to start.
  • Self-contained sections matter. An agent might retrieve just one heading’s worth of content. Write introductions and section openers that stand alone.