Why Your Scraper Gets Blocked: Browser, Proxy & Fingerprint

Your web scraper may be getting blocked because of request volume, an unaccepted network route, browser or protocol signals, or an access rule. It may also be logged out or extracting a page that has not rendered yet. Start by identifying which response and page state you actually received. A new proxy cannot fix an expired login, and a browser cannot grant permission the account does not have.

For an authorized collection task, the useful sequence is: preserve one failure, classify it, change one relevant variable, then verify the original data requirement. Treat a CAPTCHA or explicit refusal as a review boundary. Keep the failed observation even if the next attempt works.

Is your scraper actually being blocked?

First separate the request path from the extraction path. Did the connection reach the destination? Did the intended page arrive? Did the fields you need appear? Those are three separate checks.

These are starting points, not a detector for a website's internal scoring. MDN defines 403 as a refusal to fulfill an understood request. It does not mean the site has conclusively identified your IP or browser fingerprint. A 407 concerns the intermediary proxy, which is a different place to investigate.

Some providers expose a stronger signal. Cloudflare documents the cf-mitigated: challenge response header for its Challenge Pages, which return HTML even when the request expected another content type. Use that documented challenge signal where present. Its absence does not rule out every challenge or access control on the web.

What should you record before changing anything?

Save enough evidence to compare like with like: timestamp, requested and final URL, status, content type, redirect destination, relevant response IDs, browser/client version and expected output. Add a non-secret account alias, session age, approved route label, worker count and total request rate. A redacted screenshot can distinguish a login prompt from an empty result set much faster than another exception stack.

This illustrative incident record describes what to retain. It is not an observed run, and its placeholders are deliberately not credentials.

{
  "run_id": "diagnosis-001",
  "requested_url": "https://example.com/catalogue",
  "final_url": "<redacted final URL>",
  "status": "<observed status>",
  "content_type": "<observed content type>",
  "page_kind": "unknown",
  "account_alias": "approved-reader",
  "route_alias": "approved-egress-a",
  "expected_fields": ["record_id", "title", "source_url"],
  "accepted_records": null,
  "change_under_test": null
}

Then write a hypothesis that the next observation could disprove. For example: “The fresh context has no valid login, because the final page is sign-in.” Restore that account through the approved login path and check the same read-only target. Changing the IP, User-Agent, cookies and pacing together may produce a working page, but it leaves you unable to explain which change mattered.

Remove authorization headers, cookie values, tokens, signed URL parameters and personal page content before sharing artifacts. Preserve only the fields needed for diagnosis, with an access and retention policy.

What did the Airbnb diagnosis actually find?

The screenshots and saved run artifacts supplied for this article document two read-only Airbnb inspections on October 20, 2026. Both searched London for December 8–10, two adults and an entire home. Both reached search results and the first listing detail. Neither recorded a CAPTCHA, access refusal or required sign-in for the captured task. These were individual diagnostic runs, not repeated reliability trials.

The fresh headed Playwright context requested en-US at 1280 × 800. Airbnb switched to its Singapore domain and displayed English (SG) with SGD prices. The first saved listing, Premium Studio 12 @ The Orange Door Little Venice, matched its detail-page identity and displayed price for the selected two-night stay.

The run still had failed UI steps. One call log shows an informational dialog intercepting pointer events, an unstable element and a detached DOM node. Those messages point to page interaction and readiness, rather than proving an IP or fingerprint block. The run retained its errors and corrections. Application-level navigation retries were disabled; Playwright's built-in actionability polling remained enabled.

Observed call-log excerpts:
subtree intercepts pointer events
element is not stable
element was detached from the DOM, retrying

Four IDs overlapped between the two original five-card sets. Sessions, currencies, browser builds, viewport sizes and observation times differed, so the runs do not establish why rankings differed or which product is more reliable. Neither proxy quality nor fingerprint detection was tested. Search/filter changes used client-side routing, and the Playwright records lack navigation-response metadata for those transitions; ego(lite)'s HTTP status and header measurements were unavailable. A working page is the observed result, not a complete network diagnosis.

Why do CAPTCHA and browser challenges appear?

A challenge means the site requires another check before continuing. Rate, network reputation, browser capabilities, session history and configured security rules can all be relevant. From a screenshot alone, you generally cannot tell which combination triggered this particular check. If you operate the site, correlate its security logs and request ID with the scraper timestamp instead of guessing from the artwork.

For a third-party site, pause the job. A person can review the challenge through the site's permitted flow, or request an API, export or approved access route. Do not keep retrying the challenge, outsource its solution automatically, or interpret a successful manual visit as permission for unrestricted collection.

Also inspect ordinary browser failures. A script-blocking extension or broken JavaScript can prevent a legitimate page from completing its normal flow. Cloudflare's challenge troubleshooting guidance discusses browser and network compatibility. That supports checking a supported, normally configured browser; it does not justify disguising the client to defeat a refusal.

What does browser fingerprint consistency mean?

A browser fingerprint is a combination of observable characteristics, rather than one magic identifier. MDN's definition includes browser and device configuration signals. A copied User-Agent string does not turn an HTTP library into the browser named in that string, supply its JavaScript behavior or reproduce its connection implementation.

Transport fingerprinting is a separate layer. Cloudflare's JA3/JA4 documentation describes fingerprints derived from how a TLS client initiates a connection. A request header does not change that handshake. If you own the destination, inspect the relevant security event; a scraper-side 403 alone cannot tell you that a TLS rule caused the refusal.

For diagnosis, record the actual client and its configuration. Check whether custom middleware is overriding headers, whether the expected locale and viewport were applied, and whether an extension changes the page. Let a maintained browser generate its normal browser-controlled headers. Give an API client the documented headers for that API. Avoid a hand-built mixture that claims one client while running another.

Keep consistency checks separate from permission checks. A normal Chromium session can still receive a refusal. Patching browser properties, forging fingerprints or impersonating a search engine would also destroy the clean baseline you need to understand the incident.

Is the proxy failing, or is the destination refusing access?

A proxy adds another connection and authentication boundary. Check its connection error or response before attributing everything to the destination. Record the approved exit region and a route alias, and ask the provider or network administrator about an outage, authentication failure or routing restriction. Never log the proxy URL if it embeds a password.

Distinguish the exit network from the session policy. Datacenter proxies use hosting-network exits; residential or ISP-labelled offerings use ISP-addressed exits, with sourcing that needs verification. Mobile offerings use carrier-network exits. Separately, a static or sticky assignment aims to retain an exit for its documented lifetime, while a rotating policy changes it. Check the actual provider contract rather than assuming any label guarantees continuity or acceptance.

Choose a route for an authorized requirement such as organization egress or regional QA, with documented provenance. Do not buy a different proxy merely because a response says access is forbidden. Where possible, an operator-approved allowlist or official data interface gives a clearer contract than trying more exits.

Rotation is a routing policy, not a repair for every block. During a stateful workflow, retain the expected approved route unless its contract requires otherwise. If authorized failover changes the route, checkpoint first and revalidate the account and page afterward. Do not rotate to evade a denial or distribute requests to get around a limit.

How should rate limits, sessions and cookies work together?

Set a budget for the whole workload. Five workers each obeying a local delay can still exceed one shared account or site limit. Count retries, pagination and background work against that budget. Cache permitted responses, deduplicate queued URLs and request only what the task needs.

When a 429 response includes Retry-After, honor it across the affected workers. The value can describe a delay or a date. If waiting would exceed the job deadline, save progress and defer or stop. When no delay is supplied, use the operator's documented limits and a bounded backoff policy. There is no universal safe requests-per-second number for all sites.

Cookies need a separate policy. Keep state within the intended account and task, check expiry through an observable authenticated page, and pause for reauthentication when needed. Do not copy raw cookies from a personal profile into logs or a shared worker pool. Playwright's authentication guidance treats stored authentication state as sensitive and notes that it can expire. Saving state does not guarantee that the server will accept it later.

If your problem is specifically how to retain the intended login, continue with persistent browser sessions. If the expected page loads but its fields changed, move to scraper breakage after page changes. That distinction keeps an extraction bug from turning into an expensive proxy investigation.

When does a visible signed-in browser help?

It helps when you need to see why an authorized interactive task differs from a fresh scraper: wrong account, missing login, consent dialog, unapplied filter or incomplete rendering. ego(lite)'s official site describes reusing an everyday Chromium login environment, Profiles and extensions. Its Space documentation describes a dedicated task workspace. Confirm the selected Profile and the actual login inside that Space before reading any account data.

In the supplied ego(lite) run, Codex used the named Space “Scraper diagnosis: Airbnb session inspection.” The saved observations record a Log out menu item before and after handoff, without opening a private account page. That is evidence of signed-in site chrome, rather than an independent server-side identity audit. The fresh Playwright context also accessed the public results, so this task did not demonstrate a login-access advantage.

The quick start describes watching the task and taking over. Here, the agent handed control to the user for an operator-review drill. The native bar changed to “You're in control” with a Return to agent button. The user explicitly replied “hand back,” and the agent resumed the same Space with a fresh observation. No authentication or security challenge occurred, and the user reported no review findings; the handback is not a separate human validation of the data.

The run also retained friction: an early minimized-window capture failed, and clicking the first listing anchor timed out when an image intercepted pointer events. The agent took a fresh Snapshot and navigated through ego(lite) to the URL observed on that card. The later native frames establish the visible portion of the task; they do not prove that earlier minimized operations were visible. This inspection does not establish anti-bot bypass, fewer challenges or a replacement for a proxy.

For an approved API or export, use that simpler route. For unattended server collection, choose a supported runtime and an explicit access contract. The login-wall guide covers that access choice in more detail.

How do you verify that the repair worked?

A repair is complete when the original task produces accepted source data under the recorded conditions. “The request stopped throwing” is too weak. A smaller exception count can simply mean the extractor accepted an error page as an empty catalogue.

• Confirm identity: expected origin, route, account and filters, with no unexpected login or challenge page.
• Check the content: required fields, record IDs, source URLs, duplicates and the meaning of zero results.
• Verify a source sample: compare records from different pages against the visible originals; retain the check separately from extraction.
• Repeat within the approved budget: preserve the initial failure and the changed configuration. One success does not establish a general recovery rate.
• Monitor the next run: separate rate-limit events, auth pauses, challenges, transport errors and rejected data. Alert on false success as well as exceptions.

A price-extraction correction is not a price change

The ego(lite) parser initially saved ¥52,810 JPY for listing 51439688. Its initial Snapshot already contained both that crossed-out comparison price and the active total of ¥49,163 JPY. The run preserved the frozen output and stored a separate correction. Comparing ¥52,810 with the later active total would have falsely reported repricing. The corrected initial total matched the reload observation.

The final supplied report retained all five original IDs, order and active prices after one ego(lite) search reload. “5/5” here counts tracked listing identities in one inspection; it is not five successful trials or a reliability rate. The “4/5” overlap concerns only the two original top-five sets, not every available listing. No conversion or equality between the SGD and JPY prices is asserted.

For example, in a hypothetical catalogue task a request follows a redirect and ends on a sign-in page with status 200 and zero cards. The first repair is authorized session restoration. Afterward, confirm the catalogue route and its actual card identities and fields. If login works but collection is still refused, that is a separate access decision requiring operator review.

FAQ

Why is my scraper blocked even with a proxy?

The proxy changes part of the network path. It does not supply account permission, renew a session, execute a missing application flow or remove a rate limit. Inspect the response and page before deciding that the proxy is the cause.

Does a 403 prove the site detected automation?

No. It proves refusal, with the cause depending on the responding system. Retain the body and request ID, check access policy and ask the operator where possible. Do not infer a specific fingerprint rule from the status alone.

Should I rotate IPs for every request?

Follow the authorized route and session contract. Per-request rotation can make a stateful diagnostic run harder to interpret. It is not a substitute for respecting account limits or resolving denied access.

Why can I see the page manually while my scraper gets no data?

Compare account, session, final URL, rendered content and filters first. Manual access and an empty fresh context are different starting states. If those match, inspect extraction readiness and field validation before changing the network route.

For the execution-layer decision, compare ego(lite) and Playwright. Choose the route that fits the authorized task and gives you enough evidence to inspect its result.