Detecting the Detection: Finding Out Which Signal Blocked Your Scraper

Your scraper got a 403. Now what? The internet’s default answer is folklore: “clear your cookies”, “rotate your IP”, “use a real browser”, “slow down”. Sometimes that advice is right and sometimes it wastes a day, because a block caused by the word HeadlessChrome in your User-Agent and a block caused by your IP’s reputation have completely different fixes. Guessing which one bit you is not engineering - measuring it is.

This article is about the measurement. In June 2026, a three-person team at the University of Bamberg published Detecting Bot Detection: Prevalence, Techniques, and Implications for Web Measurement Research (Gundelach, Muehlhauser, Herrmann), a study that did at scale what most of us do by superstition: it crawled the Tranco Top 10,000 sites in four browser configurations, 40,000 page visits total, and worked out - with a controlled experiment - which signal actually caused each block. The answer was not the one the scraping ecosystem assumed.

We will dig into what the study found, then build a pocket-sized version of its methodology you can run against your own targets: a differential prober that visits one URL in six client configurations, each differing from the next in exactly one signal layer, and reads the pattern like a diagnostic tree. Every output in this article is a real run from October 28, 2026, on a single UK residential connection: Playwright 1.64.1, Chromium 156.0.8078.4, Firefox 157.0.

This is the fourth article in our anti-detection series, after coherent identity spoofing with Camoufox, the Patchright driver teardown, and Cloudflare’s Adaptive Intelligence economics. Those articles were about not being detected. This one is about knowing when, and how, you already were.

What the Study Actually Measured

The Bamberg group’s framing is aimed at academic web measurement, not scraping, which is what makes it refreshing: there is no product being sold and no bypass being promoted. Their question was scientific - when researchers crawl the web with automated browsers, how much of the web silently goes missing?

The answer starts with a literature audit that should embarrass the field. Of 81 crawling-based papers published between 2020 and 2025 at top-tier venues (USENIX Security, IEEE S&P, CCS, NDSS, WWW, IMC, PETS), 83% never mention bot detection or blocking at all, and only 5% quantify their blocking rate. Papers addressed an average of 2.4 of the seven detection-related documentation criteria the authors checked. Ten percent did not touch a single one. In other words: a large fraction of published web measurements may be describing a systematically filtered version of the web, and the field mostly does not know or report which fraction.

Then the infrastructure reality. The team resolved the domains of the Tranco Top 1 Million and mapped them to network owners: Cloudflare alone accounts for 40.0% of AS-matched A records, Amazon 16.9%, and the top five providers 63.5%. Most of the web is served by a handful of companies whose products include default-on bot detection. Any crawling study - or scraper - is walking through their checkpoints whether it acknowledges them or not.

The main event was the scan: the top 10,000 Tranco sites, each visited in four conditions - Chromium headless, Chromium headed, Firefox headless, Firefox headed - from a single Hetzner datacenter in Nuremberg, in late February 2026. All four conditions used the same automation framework (Playwright 1.57) with no stealth measures, so any difference between conditions isolated the browser’s own signals, not the tooling around it. A visit was classified as a soft block when the server answered HTTP 403, 429, or 503; screenshots and response bodies served as ground truth for what kind of block it was.

The headline numbers:

One configuration is a clear outlier. Vanilla Chromium headless - what half the scraping tutorials on earth launch - is blocked at more than double the rate of the other three. Among the 6,936 Chromium sites where both display modes completed, 772 were blocked exclusively in headless mode; exactly 14 were blocked exclusively in headed mode. Firefox, whose paired headless/headed difference was 15 sites versus 12, barely notices the distinction at all.

That asymmetry alone would be a nice anecdote. What elevates the paper is that the authors did not stop at correlation.

The Cause Was in the Headers

Chromium’s headless mode does not just behave differently - it introduces itself differently. Three header-level tells separate it from the headed build:

• The User-Agent string contains the literal token HeadlessChrome instead of Chrome.
• Playwright’s default headless build ships the full build number in that token (143.0.7499.4) while headed Chromium reports a reduced form (143.0.0.0) - an extra uniqueness signal.
• The sec-ch-ua Client Hints header - which Chromium attaches to every request, and which Firefox does not implement at all - includes "HeadlessChrome" as a named brand.

To turn that correlation into causation, the team re-crawled the 798 sites that had blocked only their headless Chromium, with exactly one change: the HeadlessChrome tokens were patched to headed values, matching the headed configuration’s headers byte for byte. Of the 784 sites that responded, 590 (75%) let the same browser, same engine, same IP, same everything else, straight through. The remaining 194 (25%) held the block, meaning a deeper layer - JavaScript environment checks, behavioral scoring, TLS fingerprinting - was doing that work.

Three quarters of the headless-only blocking on the modern web is, measuredly, two strings in your HTTP headers. If you remember one number from this article, make it 75%.

The study’s attribution of all 1,554 soft-blocked sites, by the way, is worth copying as a classification scheme: 59.1% carried an identifiable vendor signature in the block page (Cloudflare challenges dominate at 792 sites, then DataDome 65, PerimeterX/HUMAN 26, generic CAPTCHA interstitials 23, Imperva/Incapsula 13), 22.7% blocked condition-dependently in ways only bot detection explains, and 18.1% blocked every condition uniformly - the geo-restrictions, ASN bans, and misconfigurations of the world. Combined: 82% of observed blocks were bot detection, 18% were something else wearing the same status code.

Two more findings matter to anyone who scrapes. First, blocking is provider-driven, not site-driven: sites behind Cloudflare blocked at 37.0% and Akamai at 26.4%, while providers without default-on detection ranged from 0 to 16%. Popularity rank had no meaningful correlation. The checkpoint is at the CDN, so the checkpoint is wherever the CDN is. Worse, Akamai’s blocks came back as generic error pages 103 times out of 110 - no vendor branding, indistinguishable from a broken server without body inspection. Second, the detection infrastructure is far ahead of the blocking: 34% of sites read navigator.webdriver, and 46% probe for at least one automation-framework artifact - properties like window._selenium, callPhantom, or __nightmare that have no reason to exist in a legitimate browser and no reason to be read by a legitimate page. Cloudflare’s Turnstile is now the sixth most deployed third-party script on the entire Tranco Top 10K, ahead of most analytics libraries. Most sites that can detect you currently choose not to block; that is headroom, not comfort.

The Signal Stack

Put the findings in a frame and the methodology writes itself. Every request a scraper makes is scored on five layers, roughly in this order:

1. IP reputation - your exit address’s class, history, and per-site learning state.
2. TLS and HTTP/2 fingerprint - the shape of your handshake, independent of any header.
3. HTTP headers - User-Agent, Client Hints, header order. The layer with the 75% finding.
4. JavaScript environment - navigator.webdriver, window.chrome, plugins, CDP artifacts, the whole surface our Patchright article catalogued.
5. Behavioral - mouse paths, typing cadence, navigation rhythm. Not visible to static probes at all.

Attribution is the process of finding the deepest layer that changed your outcome. You do it by changing exactly one layer at a time and watching the status code flip - which is precisely what the study’s four conditions and spoofing follow-up did, and what our PoC now does in six.

The PoC: A Pocket-Sized Replication

PoC repo: https://github.com/Hunt-Benito/detecting-the-detection-which-signal-blocked-your-scraper

Four scripts, standard venv:

$ python3 -m venv venv && source venv/bin/activate
$ pip install playwright httpx
$ playwright install chromium chromium-headless-shell firefox

Step 1: the confession table

Before probing anyone, know what you are wearing. 01_headers_exposed.py spins up a local echo server on 127.0.0.1 and points four browser configurations at it, so you can see your own request headers with zero third parties involved:

$ DISPLAY=:0 python 01_headers_exposed.py --display :0

--- chromium headless shell (Playwright default)
    user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/156.0.8078.4 Safari/537.36   <-- automation tell
    sec-ch-ua: "Not:A-Brand";v="8", "Chromium";v="156", "HeadlessChrome";v="156"   <-- automation tell
    accept-language: None
    navigator.webdriver: True

--- chromium full binary, HEADED (the human baseline)
    user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36
    sec-ch-ua: "Not:A-Brand";v="8", "Chromium";v="156"
    accept-language: en-GB,en-US;q=0.9,en;q=0.8
    navigator.webdriver: True

--- firefox headless (no Client Hints support)
    sec-ch-ua: None
    navigator.webdriver: True

(Output trimmed to the interesting rows; the script also prints the full-binary headless configuration, which still carries HeadlessChrome/156.0.0.0 in its User-Agent but a clean sec-ch-ua.) The default Playwright experience announces itself three ways before page JavaScript runs: the UA token with a full build number, the Client Hints brand, and a missing Accept-Language. Note what does not change across any configuration: navigator.webdriver is True everywhere, headed included. Hold that thought - it becomes the punchline of step 2.

Attention! The headed configuration requires a display. On a desktop machine --display :0 is enough; on a server, run under xvfb-run. Playwright cannot give you a headful browser out of a truly headless box.

Step 2: the deterministic lab

Third-party sites change their blocking state between requests - you will watch that happen in step 3 - so before going online we validate the attribution logic against a target whose behavior we control. 04_detection_lab.py is a stdlib-only server that plays a simplified detection vendor with three enforcement modes. In header mode it does one thing: if User-Agent or sec-ch-ua contains HeadlessChrome, it serves a 403. That is the study’s dominant blocking mechanism, distilled to a handful of lines of Python.

Run the lab and the differential prober against it:

$ python 04_detection_lab.py --mode header          # terminal 1
$ python 02_probe.py http://127.0.0.1:8980/ --display :0   # terminal 2

=== http://127.0.0.1:8980/
config                   status   blocked  vendor             note
plain-http               200      False    -
plain-http-chrome        200      False    -
shell-headless           403      True     -                  UA carries HeadlessChrome
shell-headless-spoofed   200      False    -                  navigator.webdriver = true
chromium-headed          200      False    -                  navigator.webdriver = true
firefox-headless         200      False    -                  navigator.webdriver = true
  attribution:
   - Header level: patching UA + sec-ch-ua lifted the block - the HeadlessChrome tokens were
     the trigger (the dominant cause in arXiv:2606.14525).
   - Engine asymmetry: Chromium and Firefox headless were treated differently - Client Hints
     absence on Firefox changes the header surface a defender can score.

The shell-headless-spoofed configuration is the star: the same headless shell, launched with a CDP Network.setUserAgentOverride that patches the User-Agent and the Client Hints brands to the headed values (verified against the local echo server - the outgoing sec-ch-ua really is clean, not just the UA string). It walks through the exact gate that stopped its identical twin. One layer changed, the outcome flipped: that is attribution.

Now the layer underneath. Restart the lab in js mode. The server answers 200 to everyone, then a client-side script - modeled on what the study’s instrumentation observed real vendors doing - checks navigator.webdriver and the Client Hints brands, and on detection replaces the page content. The status code never changes:

$ python 04_detection_lab.py --mode js

=== http://127.0.0.1:8980/
config                   status   blocked  vendor             note
plain-http               200      False    -
plain-http-chrome        200      False    -
shell-headless           200      True     -                  in-page JS block: navigator.webdriver, userAgentData.brands:HeadlessChrome, UA-token:HeadlessChrome
shell-headless-spoofed   200      True     -                  in-page JS block: navigator.webdriver
chromium-headed          200      True     -                  in-page JS block: navigator.webdriver
firefox-headless         200      True     -                  in-page JS block: navigator.webdriver
  attribution:
   - JS environment level: the in-page check flagged every browser configuration, headed
     included - navigator.webdriver is true in all Playwright conditions, so headful
     automation is not a human baseline.
   - Note: raw HTTP clients passed with HTTP 200 - they never execute the page script, so
     a JS-layer block is invisible without a browser (the study's content-degradation
     blind spot).

Read the note column as a stack trace of the detection: the default shell was flagged three ways; the spoofed shell - headers now clean - is flagged by exactly one, navigator.webdriver; headed and Firefox likewise. The prober surfaces this as a content degradation block (status 200, body replaced) because that is exactly what it is - and that is the honest blind spot the Bamberg authors list among their own limitations: their scan classified explicit 403/429/503 responses only. Blocks that hide inside HTTP 200 are invisible to log-file analysis, and only a browser that executes the script can even see them. Your attribution toolkit needs both.

The punchline from step 1 lands here: headed is not human. Every Playwright configuration, headful display included, reports navigator.webdriver === true. The study’s headed conditions approximated a human baseline but, as the authors state plainly, did not replicate one - and on a site that reads that one property, “just run headful” fixes nothing. Closing that specific gap is Patchright’s --disable-blink-features=AutomationControlled territory, covered in our previous article.

Step 3: the real world, with its moving parts

Now point the prober at something you do not control. We used four targets on October 28, 2026: bot.sannysoft.com (a public detection test page that never blocks - the control), our own site (self-hosted nginx, no CDN checkpoint in front), nowsecure.nl (the community’s classic Cloudflare challenge guinea pig), and datadome.co - the security vendor’s own homepage, which runs their product and makes a legitimate, explicitly public test target. One visit per configuration, single page, no interaction.

The control, our site, and nowsecure.nl: six passes each, nothing to attribute. Our own site passing is the finding in miniature - no CDN, no checkpoint, nothing to trip - while from a quiet residential IP making single polite visits, even the loud headless shell walked through Cloudflare’s gate at nowsecure.nl too. Your blocked scraper and our clean pass are the same software; the difference is context, which is precisely why attribution matters.

datadome.co was the interesting one:

=== https://datadome.co/
config                   status   blocked  vendor             note
plain-http               403      True     DataDome
plain-http-chrome        200      False    DataDome
shell-headless           403      True     DataDome           UA carries HeadlessChrome
shell-headless-spoofed   403      True     DataDome           navigator.webdriver = true
chromium-headed          403      True     DataDome           navigator.webdriver = true
firefox-headless         403      True     DataDome           navigator.webdriver = true

Every real browser blocked, headed included, headers patched or not - while a raw Python HTTP client wearing Chrome’s headers strolled through. That inverts the folklore: the “fake” client passed and every genuine browser failed. Re-run minutes later, the same raw client scored 403 twice in a row, on both HTTP/1.1 and HTTP/2. DataDome’s gate did not sit at one layer we could name; it sat below every layer we could patch, and it moved between requests. That is an honest attribution outcome too: “deeper than headers, stateful, IP- or fingerprint-scored - and volatile.” The verdict you cannot explain is still infinitely more useful than a guessed one, because it tells you which fixes not to waste time on. Header spoofing would have fixed nothing here. Neither would headful mode.

The IP layer is where “which exit am I on” becomes a measurable question rather than a prayer. The prober accepts a --proxy flag, so we interleaved the same header-spoofed raw client against datadome.co - direct connection versus a UK mobile-carrier exit (the 4G/5G proxy pool we run at SimplyProxies, our own service), four seconds apart:

round   direct    mobile-exit
1       403       200
2       403       200
3       403       200

Same client, same headers, same target, same minutes: the residential IP was flagged and the mobile exit was clean, three for three. That is what an IP-reputation verdict looks like when you measure it instead of assuming it - and it is also why mobile carrier exits specifically survive reputation systems that torch datacenter ranges and, as here, occasionally one residential household.

Why the difference? Exit-class economics. A datacenter address belongs to one tenant, its ASN announces it as hosting, and blocking it costs the defender nothing. A residential address carries one household - collateral damage of a handful of people. A mobile-carrier address is a different object entirely: a carrier-grade NAT (CGNAT) funnel where thousands of real subscribers - people checking maps, streaming, paying bills - share one public IP. A hard block there takes innocent bystanders down by the thousand, and defenders know it: Cloudflare publishes a dedicated detection method for CGNAT whose stated purpose is softening enforcement against shared mobile addresses to reduce collateral effects. The mobile exit in our A/B did not pass because it was “stealthier”. It passed because blocking it is expensive.

Caveat: none of this makes mobile exits magic, and our A/B is one target on one afternoon. Reputation is per-IP and dynamic: a specific carrier address that has been abused gets flagged exactly like anything else, which is why in practice the property that usually matters is rotation - a fresh exit per session rather than a sticky, burned-in one. The lesson is not “buy mobile IPs”. It is that when your differential matrix lands on “deepest layer, IP state”, the exit class is a variable you can change and measure, exactly like every other layer in the stack.

One practical note from the same experiment: Chromium refuses authenticated HTTPS and SOCKS5 proxies through Playwright’s proxy option (ERR_PROXY_AUTH_UNSUPPORTED, ERR_SOCKS_CONNECTION_FAILED), so for the browser configurations you need an unauthenticated local forwarder in front of an authenticated upstream; the raw-client leg works directly.

Step 4: honeypots - catching the probe in the act

The study’s most elegant trick deserves its own tool. Their instrumentation planted honeypot properties - names like window._selenium and callPhantom that no legitimate page would ever read - and logged anyone who touched them. Reading those names has exactly one purpose: checking whether an automation framework is present. 03_honeypot.py plants seventeen of them before page scripts run and records every access, including the script that performed it.

Against our lab in js mode - whose detection script deliberately probes the classic names - it lights up:

$ python 03_honeypot.py http://127.0.0.1:8980/

[*] 7 honeypot accesses detected

property                     accesses   family
_selenium                    1          selenium
callSelenium                 1          selenium
_Selenium_IDE_Recorder       1          selenium
callPhantom                  1          phantomjs
_phantom                     1          phantomjs
__nightmare                  1          nightmarejs
domAutomationController      1          webdriver protocol

Against the real-world targets on a passing visit: zero hits, everywhere we tried. That is consistent with the study’s own reading of the landscape - the deep multi-signal probing (Cloudflare’s scripts average 13 honeypots and exercise all eight fingerprinting categories, on the sites aggressive enough to block even headed browsers) happens on challenge paths and scored traffic, not on the polite first visit. The instrument works; the interesting traffic simply had not been triggered. Sites still probe for PhantomJS and NightmareJS in 2026, two frameworks the scraping world buried years ago - detection scripts accumulate checks like sediment, which is why the honeypot list keeps paying off long after the frameworks die.

Reading Your Own Matrix

The flowchart above is the whole discipline in one picture: change one layer, watch the outcome, name the deepest layer that flipped. A few field rules the study’s data and our runs both support:

• Run conditions in one time window. Blocks are stateful. Our identical client scored 200 then 403 from the same IP inside an hour; the study rotated IPs between batches and paired conditions per site precisely so comparisons stay fair. A config you tested yesterday is a rumor, not a control.
• The verdict ranks fixes, it does not rank tools. “Header level” means a two-line UA patch or Patchright’s driver work solves it; “JS environment” means you need the driver-layer fixes; “deepest layer, IP state” means no browser change was ever going to save you.
• A block page with no vendor branding is not automatically your fault. 103 of Akamai’s 110 blocks in the study were generic error pages indistinguishable from outages. Before you debug your stack, read the body.
• HTTP 200 is not a pass. Content degradation inside a 200 is invisible to every log-based check - the lab’s js mode exists to keep you honest about that.

What This Method Cannot Tell You

The Bamberg authors are unusually candid about scope, and our pocket version inherits every limitation while adding a few. Their scan saw one vantage point (a German datacenter) at one moment; ours saw one residential UK IP on one afternoon. They visited homepages only, classified explicit 403/429/503 only, and had no true human baseline - headed Playwright still reports navigator.webdriver === true and still speaks CDP internally, as our Patchright article showed in detail. For both studies, an observation never proves a site acted on a signal; and neither study’s static visits touch the behavioral layer at all, where the FP-Agent line of work (arXiv 2605.01247) shows the next fight is already happening: typing and scrolling dynamics that separate AI-driven agents from humans even when every fingerprint agrees. Attribution tells you which layer got you this visit. It is a snapshot, not a theory of everything.

What the Authors Recommend (and Scrapers Should Steal)

The study closes with four recommendations for measurement researchers, and every one of them doubles as scraper hygiene: document your browser engine, version, and display mode; quantify your blocking rate instead of noticing it qualitatively; document the CDN composition of your target set, because infrastructure provider - not popularity - predicts blocking; and write down your stealth measures or their deliberate absence. Their crawler, dataset, and analysis notebooks are published (Zenodo DOI 10.5281/zenodo.20334022), which is how reproduction of this kind should work.

For the working scraper, the adaptation is direct: put a six-configuration differential probe in your pipeline’s health check, log the verdict trend per target alongside your scrape metrics, and treat a drifting verdict as the early-warning signal it is. The study’s parting prediction is that as header-level signals get patched and the AI-scraping boom pushes vendors toward default-on blocking, these rates will rise. The scrapers who survive that curve will be the ones who know, on any given day, exactly which layer is rejecting them - and the ones who measure instead of guess.

Responsible Scraping

The same rules as the rest of this series: point attribution tooling at targets you own, targets that permit testing, or public test pages; keep volume to single polite visits; respect robots and terms where your situation requires it; prefer licensed APIs when they exist. Differential probing is diagnostic, not a bypass - it tells you what noticed you, and staying beneath notice is a separate (and separately documented) question.

Sources

Detecting Bot Detection: Prevalence, Techniques, and Implications for Web Measurement Research (arXiv 2606.14525): https://arxiv.org/abs/2606.14525
Full paper (HTML): https://arxiv.org/html/2606.14525v1
Study artifacts - crawler, dataset, notebooks (Zenodo): https://doi.org/10.5281/zenodo.20334022
FP-Agent: Fingerprinting AI Browsing Agents (arXiv 2605.01247): https://arxiv.org/abs/2605.01247
Playwright documentation: https://playwright.dev/python/
bot.sannysoft.com - browser automation detection test page: https://bot.sannysoft.com/
nowsecure.nl - community Cloudflare challenge test page: https://nowsecure.nl/
MDN Web Docs - Navigator.webdriver: https://developer.mozilla.org/en-US/docs/Web/API/Navigator/webdriver
MDN Web Docs - User-Agent Client Hints: https://developer.mozilla.org/en-US/docs/Web/API/User-Agent_Client_Hints_API
Cloudflare - One IP address, many users: detecting CGNAT to reduce collateral effects: https://blog.cloudflare.com/detecting-cgn-to-reduce-collateral-damage/
Hunt-Benito - The Switches That Give Away the Driver (Patchright): https://www.hunt-benito.com/blog/the-switches-that-give-away-the-driver-inside-patchright-the-undetected-playwright/
Hunt-Benito - The Fingerprint That Hangs Together (Camoufox): https://www.hunt-benito.com/blog/the-fingerprint-that-hangs-together-coherent-identity-spoofing-with-camoufox/
Hunt-Benito - The Rule That Vanishes (Cloudflare Adaptive Intelligence): https://www.hunt-benito.com/blog/the-rule-that-vanishes-cloudflares-adaptive-intelligence-and-the-new-economics-of-scraping/
Hunt-Benito - Type What You Hear (CAPTCHA bypass history): https://www.hunt-benito.com/blog/type-what-you-hear-a-history-and-evolution-of-captcha-bypass/