On this page
  1. Two kinds of crawler
  2. Understanding "reads ÷ visitors"
  3. Why one reads a lot and sends little
  4. Another angle: which pages are read, which are cited
  5. The page script only shows part of it
  6. What you can do
  7. Don't block across the board
  8. Check that crawlers follow your rules
  9. Improve the content that's already cited

Open the AI content ledger in the Bots report and many sites see a table like this (sample data):

OperatorPurposeReads per monthVisitors sentReads ÷ visitors
Operator ATraining8,200——
Operator AAnswer engine1,9003405.6
Operator BAnswer engine2,4004105.9
Operator CTraining5,600——
Operator DAnswer engine9001275
Answer engines: pages read per visitor sent backSample data
  • Operator A5.6pages
  • Operator B5.9pages
  • Operator D75pages
Training crawlers send no visitors, so they have no ratio. Among answer engines, Operator D reads more than ten times as much per visitor as the other two.

The usual first reaction: "They read all that and sent almost nothing back?"

Here's how to read the table.

Two kinds of crawler

Crawlers from AI companies do roughly one of two jobs:

  • Training crawlers download pages in bulk to train the next model. What they read becomes part of the model's knowledge, but no answer will carry your link because of it. They never send visitors back by design.
  • Answer-engine crawlers read a few relevant pages in real time when someone asks a question, summarise them into the answer and usually cite the sources. They can send visitors.

The ledger lists them separately. When comparing reading with returning, only the answer-engine rows mean anything.

Understanding "reads ÷ visitors"

The ledger's ratio is reads divided by human visitors sent back. Lower is better: less reading, more people.

In the example, the answer engines of operators A and B send about one visitor for every six pages read; operator D read 900 pages and sent 12 people, a ratio of 75.

The ratio varies enormously between sites, and there's no "normal" value. More useful comparisons:

  • Against yourself: is the same operator better or worse than last month?
  • Against other operators: among answer engines, why is one clearly worse?

Why one reads a lot and sends little

Common reasons:

  1. Your content answers the question and nobody needs to click. For definitions like "what is UTM", the AI's answer is enough.
  2. The product doesn't make source links prominent. Some put them at the end of the answer, some in a collapsed section, and click-through differs accordingly.
  3. Its users don't overlap with your business.
  4. The pages it reads don't convert — lots of archive or tag pages, for instance.

Another angle: which pages are read, which are cited

Below the ledger is "most-read pages"; in AI visibility there is "cited pages" — landing pages that actually received visitors. Compare the two and you'll often find:

  • Pages read a lot and cited a lot — your most valuable content in the eyes of AI. Look after them.
  • Pages read a lot and rarely cited — perhaps too general to be useful, or definitions that don't need a link in the answer.
  • Pages read little but cited with good conversion — give them more internal links so crawlers find them more easily.

The page script only shows part of it

Many crawlers download the HTML without running JavaScript. With only the page analytics script installed, they're invisible, and the Bots report says so at the top.

To see the full volume, add data from the server side: submit gateway or CDN access records, or regularly import Nginx or Apache logs. Both keep only records identified as bots, so nothing is double-counted with human visits. Setup is in How bots and AI crawlers are identified.

What you can do

Don't block across the board

Seeing an AI read a lot and send little, the first instinct may be to block it in robots.txt. Think it through:

  • Blocking training crawlers won't affect today's visitors, but over time models will know less about you.
  • Blocking answer-engine crawlers directly reduces the chance of being cited in answers.

It's a trade-off with no standard answer. Our suggestion: watch two or three months of data, and only restrict an operator once it's clear it reads without ever sending anyone.

Check that crawlers follow your rules

If robots.txt already has restrictions, use the crawl health report to confirm that the crawlers concerned really stopped reading those paths after the rule appeared.

Improve the content that's already cited

The most practical investment is maintaining the pages AI already cites: accurate information, clear structure, direct answers. Visitors from AI arrive with a specific question, and whether the page answers it at once decides whether they stay.

TagsAI crawlersbot trafficcontent strategy
Related featureAI VisibilityAI visibility in two ledgers: which AI assistants send you visitors and which AI crawlers read your pages — with conversions and robots.txt compliance.See how it works

Related articles

Product notes3 min read

Why bot traffic gets its own report

By default we take identified bots out of your human reports rather than dropping them or mixing them in. The three reasons behind that, and what it means for your numbers.

Read article

AI search & GEO4 min read

Visitors from ChatGPT: how to spot them, measure them and decide if they matter

More and more people ask an AI first and click through second. Where that traffic hides, how it differs from search, and how to tell how much it matters for your business.

Read article

Playbooks5 min read

Checkout drop-off: a seven-step investigation from data to fix

Plenty of add-to-carts, not many payments. A checklist to work through in order — rule out technical problems first, then find which step, which visitors and what cause.

Read article

See it on your own data

Install the snippet in five minutes and get your first report today.

Start free