Skip to content

feat(devices): name the crawlers production reads generically - #1

Merged
joelwurtz merged 2 commits into
mainfrom
feat/traffic-2026-09-21
Sep 21, 2026
Merged

feat(devices): name the crawlers production reads generically#1
joelwurtz merged 2 commits into
mainfrom
feat/traffic-2026-09-21

Conversation

@joelwurtz

@joelwurtz joelwurtz commented Sep 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

User agents the detector read only as Generic Bot, or not at all, each looked up on the web one at a time. An entry was written where an operator or a project says what the thing is. The last section says why the rest stays generic.

Bots (src/devices/bots.yml)

Health probes and monitors

user agent now reads documented at
kube-probe/1.34, /1.35, /1.35+, /1.36 kube-probe, Site Monitor kubernetes.io — liveness, readiness and startup probes (the kubelet sends kube-probe/<its own version>)
Edge Health Probe Edge Health Probe, Site Monitor Azure Front Door health probes
zabbix, Zabbix Zabbix, Site Monitor Zabbix web monitoring, ZBXNEXT-1821
Blackbox-Exporter/0.28.0 Blackbox Exporter, Site Monitor prometheus/blackbox_exporter — the default UA is Blackbox-Exporter/<version> since the RFC 9110 change
Telegraf/1.39.3 Go/1.26.5 Telegraf, Site Monitor influxdata/telegraf, http_response input
LogicMonitor SiteMonitor/1.0, LogicMonitor InternalService LogicMonitor, Site Monitor LogicMonitor — data monitored for websites
Gatus/1.0 Gatus, Site Monitor TwiN/gatus — GatusUserAgent = "Gatus/1.0" in config/endpoint/endpoint.go
yaut monitoring, YAUT monitoring, Mozilla/5.0 (compatible; Yaut/1.0; …) Yaut, Site Monitor yaut.jolicode.com — JoliCode's own URL monitor
updown.io daemon 2.11 updown.io, Site Monitor updown.io/about

Agents a platform sends on its own behalf

user agent now reads documented at
redirection-io-agent-proxy/agent-proxy-v3, redirection-io-agent/2.2.6…3.3.0 redirection.io Agent, Service Agent redirection.io agent documentation
amazon cloudfront, Amazon CloudFront Amazon CloudFront, Service Agent CloudFront — request and response behaviour for custom origins (the origin request carries Amazon CloudFront unless the real UA is forwarded)
symbolicator/26.8.0, /26.9.0 Symbolicator, Service Agent getsentry/symbolicator
Google-Cloud-Tasks Google Cloud Tasks, Service Agent Cloud Tasks — create HTTP target tasks ("the User-Agent header is set to Google-Cloud-Tasks")
Mozilla/5.0 (compatible; Cloudinary/1.0) Cloudinary, Service Agent Cloudinary — deliver remote media files
WinHttp-Autoproxy-Service/5.1 WinHTTP Web Proxy Auto-Discovery Service, Service Agent WinHTTP AutoProxy support — the GET /wpad.dat of the WPAD service
ias-va/3.3 (former https://www.admantx.com + …) ADmantX Service Fetcher, Service Agent Integral Ad Science — about IAS; matomo already reads the /3.1 string, so this entry sits at priority -1 under it
Amazon Simple Notification Service Agent Amazon Simple Notification Service, Service Agent SNS — HTTP/HTTPS headers
cert-manager-challenges/v1.21.2 (linux/amd64) cert-manager/… cert-manager, Service Agent cert-manager — ACME orders and challenges; the linux/amd64 had it counted as a person at a keyboard

Search, and the crawlers that feed an answer rather than an index

user agent now reads documented at
Mozilla/5.0 … Reflectionbot/1.0; +https://reflection.ai/bot … Reflectionbot, AI Data Scraper reflection.ai/bot — collects content to train Reflection's models
YisouSpider YisouSpider, Search bot sm.cn — Shenma's crawler, formerly Yisou
Mozilla/5.0 … KeenableBot/1.0; +https://keenable.ai/ KeenableBot, AI Search Crawler keenable.ai — its own crawler, index and ranking for AI agents (TechCrunch)
LinkupBot/1.0 (LinkupBot for web indexing; https://linkup.so/bot; …) LinkupBot, AI Search Crawler linkup.so/bot. Not LinkUp's job crawler at linkup.com, which is a different company with a nearly identical name
Mozilla/5.0 … AzureAI-SearchBot/1.0; AzureAI-SearchBot, AI Search Crawler Microsoft publishes no page of its own for it, so the entry carries the producer and no url (Rankly, botcrawl)
quillbot.com QuillBot, AI Assistant quillbot.com; the token is listed as quillbot-com in ai.robots.txt, and the entry reads both spellings

YisouSpider is the only entry in the Search bot category.

The rest of the crawlers

user agent now reads documented at
mozilla/5.0 - research.hadrian.io Hadrian, Security Checker hadrian.io — attack-surface scanning
Mozilla/5.0 (compatible; EchoboxBot/1.0; hash/…; +http://www.echobox.com) EchoboxBot, Crawler echobox.com, the address in its own string
Mozilla/5.0 (compatible; INA dlweb; +https://www.ina.fr/collecte-du-depot-legal-web) INA dlweb, Crawler INA — collecte du dépôt légal du web
Mozilla/5.0 (compatible; CorporateResearch/1.0; …; +https://climatax.org/…) ClimaTax Research Bot, Crawler climatax.org/contact/research-bot — ESG research. CorporateResearch alone would read half the research crawlers there are, so the entry asks for the address too
billigerbot/1.0 (https://www.solutions.billiger.de/de/bot) billigerbot, Crawler solutions.billiger.de/de/bot — solute GmbH's product-feed fetcher
Mozilla/5.0 … AmazonProductDiscovery/1.0; https://vendorcentral.amazon.com/… AmazonProductDiscovery, Crawler Amazon Vendor Central — amazonproductbot
Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; +http://www.leiki.com) Leikibot, Crawler leiki.com, now DoubleVerify's ad-verification crawler
SEBot-WA SEBot-WA, Crawler SE Ranking — verifying the crawler. The name looks like a search-engine bot and is not one: it is the default agent of SE Ranking's Website Audit
Mozilla/5.0 (compatible; Funnelback) Funnelback, Crawler Squiz — configuring the web crawler; that string is the documented default, comment and all
Mozilla/5.0 … SemjiBot/1.0; +http://semji.com … SemjiBot, Crawler Semji — all you need to know about SemjiBot
Mozilla/5.0 (compatible; idealo-bot inventory-image-downloader; …) idealo Bot, Crawler partner.idealo.com/de/bot
tphotobot/0.1 (+https://crawler.estidraft.com/bot) tphotobot, Crawler crawler.estidraft.com/bot — checks photographs for unlicensed use
intrinsec easm vulnerability scanner Intrinsec EASM, Security Checker Intrinsec — EASM

Load generators

user agent now reads documented at
k6-performance-test K6 k6.io. matomo already reads k6/<version>; this string has no slash, so the entry only adds the shape and reuses matomo's name rather than contradicting the rule beside it
oha/1.8.0 oha, Benchmark hatoo/oha

Clients (src/devices/clients.yml)

user agent now reads documented at
GoCityAndroid/10.2.0, /10.4.1, GoCity/39 and /48 with CFNetwork Go City, mobile app gocity.com, App Store, Play
MAXEDA/PraxisApp; Praxis, mobile app Maxeda DIY on Google Play, the app's launch
apodiscounter …, apotheke.at …, apo.com … IOS Version … apodiscounter / apotheke.at / apo.com, mobile app apo.com Group's online pharmacies, APO Pharmacy B.V. on Play. One application, named after whichever domain it sells under
ginlemon.flowerfree/6.6 build 020/true and /false Smart Launcher, mobile app ginlemon.flowerfree is Smart Launcher's Android package
Symfony BrowserKit Symfony BrowserKit, library symfony/symfony#50052 — the default agent of AbstractBrowser
Zeep/4.3.3 (www.python-zeep.org) Zeep, library docs.python-zeep.org
MWFeedParser MWFeedParser, library mwaterfall/MWFeedParser — [req setValue:@"MWFeedParser" forHTTPHeaderField:@"User-Agent"]
rss-parser rss-parser, library rbren/rss-parser — DEFAULT_HEADERS = { 'User-Agent': 'rss-parser', … }

The client and only the client. Nothing in GoCityAndroid/10.2.0 or in an Android package name
says which system it runs on, and the client axis, where a hand-written client entry goes, has
no business inventing one.

What was left alone, and why

A few were already named here; the detector they were compared against is older.

The rest name nothing a search can place, and Generic Bot or no answer at all is the honest
reading. A bot regex written on a guess turns readers into crawlers on every site it serves:

  • a crawler with no page behind it: AionBot/1.0 (Udger lists it with an unknown author and
    no URL), [SEOBot] (seobotai's own crawler
    documents a
    different string), SSI-Nutch/1.23, PatteBlancheBot/0.1, ExitChecker/1.0.0,
    netcheck-probe, WPImageProxy, KELKOO resizer, and the plain Varnish, which is not the
    Varnish Health Probe the documentation writes.
  • a scraper someone wrote for themselves, or a research string with no operator.
  • a site's own front end, application or test runner.
  • a string that names nothing at all: bare Mozilla/5.0 strings, node, app-iphone,
    webview-android and the like. The Apple networking processes already read as Safari, which is
    what matomo answers.

tests/pinned/production.rs pins both halves: one case per entry written, and a guard that the
most common of the unnamed still read as Generic Bot, so an over-wide entry added
later cannot quietly swallow them.

Tests

cargo test --release --no-default-features    38 112 passed, 0 failed
cargo test --release                          38 112 passed, 0 failed

The corpus does not move; tests/pinned/traffic.rs stays green. Every new case in
tests/pinned/production.rs was checked to fail against the database as it was.

Two entries had to be placed under rules matomo already holds, which the corpus caught:
ias-va at priority -1, below matomo's reading of the /3.1 string, and k6-performance-test
as its own shape rather than a wider k6 rule that would have renamed matomo's.

@joelwurtz joelwurtz changed the title feat(bots): name 38 crawlers and agents read from production traffic feat(devices): name the crawlers production reads generically Sep 21, 2026
@joelwurtz
feat(bots): name 39 crawlers and agents read from production traffic
bc33437 
A dump of redirection.io's traffic of 2026-09-21 holds 150 user agents production
answers with `Generic Bot`, or with the string itself. Thirty-nine of them name
something whose operator documents it: Kubernetes and Azure Front Door health
probes, Prometheus, Gatus, Telegraf, Zabbix, updown.io, Yaut and LogicMonitor
monitors, the platform agents of CloudFront, SNS, Cloud Tasks, cert-manager,
Sentry's symbolicator and redirection.io's own agent, and the crawlers of Shenma,
Keenable, Linkup, Azure AI, Reflection, QuillBot, Semji, Echobox, DoubleVerify,
billiger.de, idealo, Amazon, SE Ranking, Squiz, the INA, ClimaTax, Intrinsec and
Hadrian.

Each entry is anchored on the token its operator publishes, and carries the page
that publishes it. The rest of the dump is left generic: a scraper someone wrote
for themselves, or a research bot with no page behind it, is a guess, and a bot
regex written on a guess reads people as crawlers on every site served.

Two entries sit under rules matomo already holds. `ias-va` takes priority -1 so
matomo's `/3.1` reading of the same fetcher still answers for the version it names,
and `k6-performance-test` reuses matomo's own name rather than contradicting the
`k6/<version>` rule beside it.

YisouSpider is the one entry in the Search bot category, which moves the row out of
the log injector's tool column, so `tests/pinned/traffic/false-negatives.yml` says
why production's answer stands.
@joelwurtz
feat(clients): name the applications and libraries production read as…
212c3d3 
… nothing

Eleven more strings from the traffic of 2026-09-21, none of which the corpus holds:
Maxeda's Praxis application, the four online pharmacies of the apo.com Group that
share one iOS application and name it after the domain they sell under, Go City on
both its platforms, Smart Launcher under its Android package name, and four
libraries -- Symfony's BrowserKit, python-zeep, MWFeedParser and npm's rss-parser --
each of which sends its own name and nothing else.

The client and only the client. Nothing in `GoCityAndroid/10.2.0` or in
`ginlemon.flowerfree/6.6` names a system, so those two keep the type production gave
them: the client axis is where a hand-written client entry goes, and it has no
business inventing one.
@joelwurtz
joelwurtz force-pushed the feat/traffic-2026-09-21 branch from 4c9a615 to 212c3d3 Compare September 21, 2026 10:27
@joelwurtz
joelwurtz merged commit 433637a into main Sep 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@joelwurtz