In controlled testing published this year by the security firm cside, conventional bot detection failed to identify AI agents in 81 out of 100 scenarios. That is not a tuning problem. The tools were looking for the wrong thing, because an agent driving Chromium with a genuine user agent string, a real TLS fingerprint and a residential IP address is, by every signal a firewall inspects, a person.
This has arrived faster than most teams have noticed, and it lands on the parts of an application that are expensive to serve and awkward to protect: authentication, search, cart, checkout and any reservation or availability endpoint. Those were rate limited for human pacing, and human pacing is no longer the shape of the traffic.
Why the existing defences cannot see this
Bot detection was built on a reasonable assumption: automated traffic looks mechanical at the network and identity layers. Scripted clients used unusual user agents, odd header ordering, datacentre IP ranges and TLS stacks that did not match the browser they claimed to be.
An agent operating a real browser has none of those tells. It is a real browser. The headers are real because Chromium generated them, the TLS fingerprint is Chromium's, and if it is running on somebody's laptop then the IP address belongs to a household. Every check a web application firewall performs returns the same answer it would for the person sitting at that desk.
The uncomfortable part is that this is not going to be fixed by a vendor update. The distinguishing information is not present at the layers those tools inspect, so no amount of rule tuning recovers it. It is a category change rather than an arms race increment.
They arrive at your most expensive endpoints
General crawling has always been cheap to absorb, because crawlers read pages. Agents do tasks, and a task means the endpoints that cost you money.
A price comparison is a search query, several product fetches and often an add to cart to see delivery options. A booking check hits availability. An account summary means a login. Each of those is orders of magnitude more expensive than serving a cached page, and each was capacity planned against a human arrival rate.
We have seen this present first as an unexplained load pattern rather than as a security alert: a search endpoint whose 95th percentile latency drifts upward with no growth in signups, or a rate limiter that starts tripping for a handful of genuine customers because the limit was set per IP and one household now generates a hundred requests a minute.
The diagnostic is usually straightforward once you look. Requests that arrive in tight bursts with no intervening asset loads, sessions that reach a product page without ever loading a font or an image, and conversion funnels where a step that normally takes forty seconds takes two.
Blocking is the wrong default
The instinct is to treat this as abuse and stop it. That instinct is expensive, because a meaningful share of this traffic is your customer, working through an intermediary.
When somebody asks an assistant to compare three suppliers and it visits your site, blocking it removes you from the comparison. You do not appear as blocked, you appear as absent, and the recommendation goes to whoever answered. That traffic already converts at a rate well above search, which we looked at in why AI referrals convert so much better, so the cost of a false positive here is high and completely invisible in your analytics.
There is a real abuse case too, and it deserves separating out. Credential stuffing at browser fidelity, scraping your entire catalogue for a competitor, and automated inventory hoarding are genuine problems that got easier this year. The mistake is treating one policy as covering both, because the response to a customer's assistant and the response to a scraper are opposites.
Behaviour is the only signal left
If identity and network signals are exhausted, what remains is what the session does, and this turns out to be surprisingly legible.
Humans scroll before they act, move a pointer along imprecise paths, pause to read, and load the images and fonts on the page because their browser renders them. Agents frequently read the document object model directly in a burst, act without scrolling into the viewport, follow deterministic pointer paths if they use one at all, and leave artefacts from the browser automation protocol that drives them.
None of these is proof on its own and all of them are evadable by a determined operator. That is fine, because the goal is not certainty. The goal is a confidence score good enough to route on, and behavioural signals get you there for the overwhelming majority of ordinary agent traffic, which is not trying to hide.
The practical version is to score rather than to classify, and to attach different consequences to different scores. High confidence abuse gets blocked. Probable agent gets a different rate limit and a cheaper response path. Everything else proceeds untouched.
Build a path for them instead of a wall
The strategic answer is less defensive than it sounds. If a growing share of your visitors are agents acting for real customers, the efficient move is to give them a cheap, structured way to get what they came for.
An agent scraping your product page to extract a price is doing expensive, fragile work because you offered it nothing better. A documented endpoint that returns the same information as structured data serves that request at a fraction of the cost, more reliably, and on terms you set. You convert an adversarial interaction into a supported one.
The same reasoning applies to your content. Machine readable pricing, availability and specification data is now infrastructure rather than a nicety, for the same reason the shift in where demand comes from now rewards being extractable rather than merely visible.
Do the pragmatic version first. Structured data on the pages that matter, a stated policy on automated access, and a documented route for the traffic you want. That handles the cooperative majority and leaves your detection budget for the traffic that is actually hostile.
Your analytics are already wrong
One consequence that catches teams late: if you cannot distinguish this traffic, it is inside every number you report.
Sessions are inflated, bounce rate is distorted in whichever direction the agent behaviour happens to push it, average session duration is meaningless, and conversion rate is understated because the denominator contains visits that were never going to convert in the session you measured. Decisions get made on those numbers.
Segment agent traffic out of your reporting before you do anything else, even if your segmentation is imperfect. An approximate split is far more useful than a precise total of two different populations added together, and it usually changes what the funnel appears to be telling you.
The first time we ran that split for a client, their conversion rate moved by more than a percentage point once the automated sessions came out, in the direction that made the previous quarter's reporting wrong rather than flattering. Nobody enjoys that meeting and every subsequent decision was made on a real number.
Our web app development team now treats automated traffic as a first class user type during design rather than as an operational nuisance discovered after launch, because the architecture decisions it affects are the ones that are expensive to revisit.
Frequently Asked Questions
Can we just block all AI agent traffic?
You can, and you will block customers doing so. A large share of agent visits are people delegating a task, and blocking them removes you from consideration silently. Separate abusive automation from delegated browsing and treat them differently, because one policy cannot serve both.
Will a web application firewall handle this?
Not on its own. A firewall inspects identity and network signals, and an agent driving a real browser produces genuine ones. It remains useful for conventional attacks and it is structurally unable to see this, so treat it as one layer rather than the answer.
What is the single most reliable signal?
Behaviour, and specifically the absence of the things humans do incidentally. No scrolling before an action, no image or font loads, tightly clustered request timing, and pointer paths that are either perfectly straight or missing entirely. Combine several rather than trusting one.
Does serving structured data help or invite more traffic?
Both, and the trade is usually favourable. You will get more automated requests and each one costs you far less than the page scrape it replaces, and it returns the answer you intended rather than whatever an extractor guessed from your markup.
How do we separate agent traffic in analytics?
Score sessions on behavioural signals server side and pass the score into your analytics as a dimension, then segment on it. Perfect classification is not the goal. Even a rough split stops you averaging two unrelated populations into one misleading conversion rate.
Should our rate limits change?
Almost certainly. Limits set per IP address assume one person per address, and a household running an assistant breaks that assumption immediately. Move to per session or per account limits where you can, and set the per IP ceiling high enough that it only catches genuine floods.
Is this worth doing before it is a problem?
The measurement is, the engineering is not. Instrument first so you know what proportion of your traffic this is, then decide. Most teams discover the number is either negligible or considerably higher than they assumed, and both answers are worth having before anyone plans work.
Talk to our team if your load is rising, your conversion rate is drifting, and nobody can account for the traffic.