How bots and AI crawlers are identified

Where: Website analytics › Bots.

Default: identify and count separately

New projects put identified bots in their own report and keep them out of visitors, sessions and pageviews. The conversion and bounce rates on your overview are therefore based on people.

Detection uses 197+ rules across 5 layers: the user agent, request headers, client-side signals on the page (such as traces of automated browsers), the network the request comes from (data-centre AS numbers) and request frequency. Each verdict is "verified", "suspected" or "unverified" — unverified doesn't necessarily mean disguised.

If a record looks wrong, click Report a mistake in the report and the rule is reviewed.

What the report shows

The AI & agents tab:

• AI requests, AI agents, AI crawlers, and the visitors AI sent back.
• The AI content ledger: crawls per operator (training crawlers and answer engines separately), the visitors each one sent, and the ratio between them.
• The pages read most.

The AI visibility tab puts crawls and AI-referred visits side by side; see AI visibility.

The All bots tab lists each bot with its operator, purpose (search, AI training, AI answers, monitoring and so on), requests, pages visited, common status codes and when it was last seen. Search engines get a crawl health table as well: crawl volume, success rate, 404s and 5xx errors.

Crawling and sending visitors are different things

"AI sources" in the Sources report counts people: someone clicked a link inside ChatGPT or a similar product and arrived on your site. AI crawlers in the Bots report count programs reading your pages.

They can be completely out of step. An AI company's crawler reading thousands of your pages in a week while sending almost nobody back is common. Looking at both together is the only way to tell whether reading and returning are in proportion.

What the page script alone can't see

The page script only sees bots that run JavaScript. Many crawlers download the HTML without running scripts, so they are invisible from the page — the report says so at the top.

For the full picture, including status codes and response times, add data from the server side in one of two ways:

• Submit gateway access records — your gateway or middleware sends each request with ua and url, optionally IP, status and response time.
• Import access logs — submit Nginx or Apache lines in combined format, up to 5,000 lines per request.

Both keep only records identified as bots; human requests are skipped (skippedHuman in the response), so visitors are never counted twice. Each needs an API key with its own permission — see Server-side events API.

We don't recommend switching detection off

Settings let you choose "don't identify", which mixes bots into human data. We advise against it: once mixed in, they dilute every conversion rate and can't be separated afterwards.