On this page
  1. Why not mix them in
  2. Why not just drop them
  3. How we identify them
  4. What it means for your numbers
  5. A limitation of the page script
  6. Can I change it?

New PathClue projects have one default worth explaining: identified bots are counted separately. They're kept out of visitors, sessions and pageviews and go into a report of their own.

We argued about this default for a while. The two usual alternatives are "drop them" and "don't identify, mix everything together". Here's why we chose a third way.

Why not mix them in

The most direct reason: mixed in, every rate is distorted.

Picture a small site with 800 real visits a day, plus 300 from crawlers and monitoring scripts. Mixed together:

  • Bounce rate runs high (crawlers usually read one page and go).
  • Average time on site runs low.
  • Conversion is diluted (crawlers don't sign up or buy).
  • Sources fill up with "direct" and odd referrals.

Worse, the bot share isn't constant. One week an AI company ramps up crawling, your bounce rate jumps ten points, the team spends two days investigating landing pages, and in the end nothing had changed.

And once bots are mixed in, they're very hard to take out again.

Why not just drop them

If bots distort the numbers, why not throw them away at collection?

Because bot traffic is valuable information in its own right, and increasingly so:

  • Are search engines crawling your site normally? A sudden fall in crawling or a rise in 404s often shows up before rankings drop.
  • Which of your content are AI companies reading? How many pages training crawlers and answer-engine crawlers read, and which ones, bears directly on whether your content appears in AI answers.
  • Does that reading bring visitors back? Only by putting crawler reads next to the human visitors AI sends can you tell whether it's worth it to you.
  • Is there unusual automated traffic? Someone scraping your prices in bulk, say, or a script submitting forms over and over.

Drop the bots and you can't answer any of these.

How we identify them

There are 197-plus rules looking from 5 angles:

Five angles for spotting bots
  1. User agentLegitimate crawlers say who they are
  2. Request headersBrowsers and programs send different headers
  3. Client-side signalsAutomated browsers leave traces
  4. NetworkData centres are more suspect than home or mobile
  5. FrequencyMany regular requests in a short time

Every verdict carries a confidence level: verified (confirmed, for example by reverse lookup, to be a particular search engine), suspected, or unverified. Unverified doesn't mean disguised — just not enough evidence.

When we get one wrong, you can click Report a mistake in the report, and we review and fix the rule.

What it means for your numbers

If you come from a tool that doesn't separate bots, PathClue will show fewer visitors. That's expected: the difference is the bots. How big it is depends on your site; content sites and technical documentation usually see more, because crawlers like them.

Correspondingly, your bounce rate may be lower, time on site longer and conversion higher. We haven't made the numbers look better; they're closer to how real visitors behave.

A limitation of the page script

To be upfront: the page analytics script only sees bots that run JavaScript. Many crawlers download the HTML without running it, and those are invisible from the page.

That's why the Bots report says so at the top. For the full crawling picture, you can add access records from the server side or import access logs. That data keeps only bot records and is never double-counted with human visits.

Can I change it?

Yes. Settings offer three options: identify and count separately (default), identify and drop, or don't identify.

We recommend keeping the default. Dropping loses the information above; not identifying distorts every human metric, and once mixed in it can't be separated again.

If you have a specific reason to change it, add a date annotation first. Data after the change follows the new rule and the chart will show a step; with an annotation, nobody misreads it months later.

Tagsbot trafficproduct designAI crawlers
Related featureBot Traffic DetectionBots, AI crawlers, monitors and data-center traffic identified with 197+ rules in 5 layers and reported separately — so your numbers describe people.See how it works

Related articles

AI search & GEO4 min read

How many of your pages AI crawlers read — and why it doesn't match the visitors they send

One AI company read 8,000 of your pages last month and sent 40 people back. Is that normal? How to read the ratio between reading and returning, and what you can do about it.

Read article

Playbooks5 min read

Checkout drop-off: a seven-step investigation from data to fix

Plenty of add-to-carts, not many payments. A checklist to work through in order — rule out technical problems first, then find which step, which visitors and what cause.

Read article

Privacy & compliance4 min read

Does website analytics need a consent banner? Break it down by region and method

Whether you need a banner depends on three things — where your visitors are, what you collect and what else runs on your site. A framework to structure the question, not legal advice.

Read article

See it on your own data

Install the snippet in five minutes and get your first report today.

Start free