MCP servers for live web scraping: connect tools and verify agent data
Fetch can give an agent a page's content, while Firecrawl can return structured fields. Before using either output, check the text, author, tags and source URL against the actual page. ego(lite) helps with that review by keeping the source open in a visible browser Space, where you can follow the agent's checks and take over if a row looks wrong.
The two-page quote task in this guide shows why that review matters. Simplified Fetch content omitted tags and the first author on each page; Firecrawl's original JSON assigned the wrong source URL to all ten page-two quotes. An independent ego(lite) run read both pages and checked all twenty records after reload, confirming the source-URL errors. Its records matched the later Fetch handoff and the separately corrected Firecrawl handoff.
This guide shows how to choose and connect a scraping MCP server, retain its original responses and verify the fields before passing records to another agent. The October 16, 2026 runs and documentation checks support the examples below. The original failures, explicit corrections and browser checks remain separate so you can see how each accepted result was produced.
What does the agent need from the web?
The required operation determines which server to connect. A search tool discovers URLs; a fetch tool reads a known URL; an extraction tool returns selected fields; a browser tool interacts with the page. One server can expose several of these operations, so inspect its actual tool catalog.
The MCP tools specification defines discovery and invocation. A successful protocol response still needs application-level validation: it can contain an error, partial output or content from an unexpected page.
| Operation | Appropriate output | Acceptance question |
|---|---|---|
| Search | Candidate URLs and snippets | Did the workflow fetch the chosen source afterward? |
| Fetch | Page content or raw HTML | Is the response complete and from the intended URL? |
| Extract | Records matching requested fields | Do values and associations match the source? |
| Interact | Browser state after permitted actions | Did the intended page state change? |
Start with the narrowest tool that can produce the required evidence. Loading a browser for a static document adds state to manage; reading HTML alone cannot demonstrate an interaction with rendered controls.
Which documented servers fit these operations?
The following candidates cover different retrieval needs. Fetch and Firecrawl have recorded task evidence below. Playwright MCP is included as a documented browser-interaction option; it was not another extraction route in this experiment.
| Candidate | Documented role | Connection example | Boundary to verify |
|---|---|---|---|
| MCP reference Fetch server | Read known URLs as text or raw content | Local uvx mcp-server-fetch | Truncation, missing rendered content and lost field associations |
| Firecrawl MCP | Hosted scraping and structured extraction | Streamable HTTP endpoint | Enabled tools, response status, schema fidelity and account limits |
| Playwright MCP | Observe and interact with a browser | Local npx @playwright/mcp | Current tab and session, fresh references and extraction completeness |
The reference Fetch README documents chunked reading through start_index and max_length, plus a raw-content option. A short first response is not proof that the page contains only that much data.
The Firecrawl MCP documentation distinguishes limited keyless access, OAuth sign-in and API-key connections. Its official server repository describes scrape and structured-output tools. Read the enabled tool schema rather than assuming every account exposes the same catalog.
Playwright MCP provides browser observations and interactions. Its role is useful when the task depends on the rendered page or navigation state. For a client-specific setup, use the Codex browser guide and the Playwright MCP setup overview.
These choices concern retrieval operations. The broader AI web scraping tools guide owns the general tool landscape; this page focuses on selecting an MCP route and accepting its returned data.
How do you connect a server without confusing registration and use?
A client needs a configured transport before it can discover server tools. Stdio launches a local executable; Streamable HTTP connects to a server URL. Installing a package or saving a URL does not prove that this conversation loaded its tools.
For Codex, the documented command forms produce distinct server entries:
codex mcp add web-fetch -- uvx mcp-server-fetch
codex mcp add firecrawl-web --url https://mcp.firecrawl.dev/v2/mcp
codex mcp listChoose the route you intend to evaluate rather than registering every candidate. These are configuration examples, not commands executed during this article's drafting. The Codex MCP documentation specifies client settings and authentication options.
The Firecrawl URL above currently supports limited keyless scrape, search and parse access. OAuth uses its documented account-connection endpoint; API-key access uses a bearer token. Keep credentials in the client's secret settings or environment and out of endpoint URLs, screenshots and saved transcripts.
Inspect the loaded tool catalog, call one known public URL, and retain the actual server name, input and returned evidence. A shell wrapper, REST request or different installed connector must be labelled as that path; it cannot silently become proof that the named MCP client integration worked.
For a local Fetch route, install uvx through the official uv instructions first, then confirm that the client can launch it. A command that works in an interactive terminal may still fail if the MCP client has a different PATH. Keep stderr separate from the stdio protocol stream and preserve the startup error before changing the executable path.
What fields and provenance should the result retain?
A data contract states which observations the downstream consumer can accept. Define identity, required fields, missing-value handling and source provenance before asking a model to produce JSON.
For a two-page quote extraction, use text, author, tags and source_url. Preserve complete text and punctuation, keep the author attached to the correct card, and retain tag order. Collapse whitespace only; do not paraphrase text to make outputs agree.
Keep run provenance alongside the records:
{
"run_id": "record-the-actual-id",
"server": "record-the-actual-server",
"tool": "record-the-actual-tool",
"requested_url": "https://quotes.toscrape.com/",
"observed_final_url": null,
"retrieved_at": null,
"retrieval_status": "not-yet-verified",
"records": []
}This is a proposed record shape. Null means that the value has not been observed. If a server doesn't expose the final URL or cache age, record the limitation instead of copying the requested URL into an observed field.
A JSON schema checks structure. It can't establish that a quote belongs to its author, that a price includes tax or that a page was fetched recently. Keep the raw server response so those questions can be revisited.
Record when the client received the response separately from any source publication date or server cache metadata. A retrieval timestamp shows when the workflow observed a response; it does not establish when the origin content last changed or whether a cached response was used.
How do you verify a multi-page extraction?
The Quotes to Scrape training site supplies a bounded example with two explicit URLs: the root page and page two. The recorded runs observed ten quote cards per page and twenty unique text-and-author pairs. These were measured on the two requested URLs, rather than inferred from a merged result table.
Call both pages explicitly. Do not assume a scrape call follows pagination or that a model's merged table includes the second page. Retain each response before normalizing records.
The checks should expose incomplete results:
| Check | Evidence | Failure handling |
|---|---|---|
| Page coverage | Both requested pages and their per-page counts | Mark a missing page as incomplete |
| Record identity | Unique text-and-author pairs | Report duplicates without silently dropping them |
| Required values | Full text, author, tags and exact source URL | Preserve missing values and reject acceptance |
| Association | Each author's name and tags checked within its card | Revisit the source before changing the record |
| Truncation | Continuation indicators and complete response coverage | Read remaining chunks or report partial content |
| Source agreement | At least ten distributed manual spot checks, including a long quote | Preserve every disagreement with the original response |
Compare parsed records by identity after independent collection. Different JSON key order or indentation is not a data disagreement. A generated result viewer helps review records, but the source pages and real MCP invocation establish where they came from.
For the reference Fetch tool, this illustrative input requests the beginning of a known page. Inspect the actual loaded schema before calling it:
{
"url": "https://quotes.toscrape.com/",
"max_length": 5000,
"start_index": 0,
"raw": false
}If the response reports truncation, retain that chunk and request its indicated continuation offset for the same URL and content mode. Record every range and assemble the content in order. Do not treat an arbitrary second 5,000-character request as proof that all intermediate content was read. If a continuation cannot be obtained or source content changes between chunks, report the incomplete coverage and repeat collection only as a separately recorded attempt.
What did the two-page MCP task actually return?
The task required text, author, ordered tags and the exact source page for each quote. All successful retrieval routes returned ten cards from each page. Acceptance changed when the four fields were checked together: the native Firecrawl structured result failed ten source-URL checks, while later local correction produced a separately labeled complete handoff.
| Route / variant | Observed collection | Accepted records | Finding |
|---|---|---|---|
| Later Fetch run | Four native calls: simplified and raw content for each URL | 20 / 20 after local parsing of retained raw HTML | Simplified content omitted tags and each page’s first author; raw HTML retained them |
| Firecrawl native JSON | Two native scrape calls; 20 returned identities | 10 / 20 under the full four-field contract | All ten page-two records incorrectly used the homepage source_url |
| Firecrawl corrected handoff | Same retained calls; no additional retrieval | 20 / 20 after explicit local correction | Changed ten source URLs using the matching request and native metadata |
| Independent ego(lite) reference | DOM collection and reloads in a visible Space | 20 independently verified records | Matched later Fetch and corrected Firecrawl; confirmed the ten native URL errors |
Fetch: raw HTML recovered fields missing from simplified text
Run fetch-20261016T075001Z used four native reference Fetch calls. Each page was requested once in simplified mode and once with raw content enabled. Simplified responses omitted all tags and the first card’s author on each page. The raw HTML retained the card structure, so a local parser could associate each full quote, author and ordered tags before producing the four-field records. The resulting JSON is locally derived output, rather than a structured dataset returned directly by Fetch.
The run recorded mcp-server-fetch 2026.10.10 and active Codex client 0.162.0-alpha.17.2. All twenty records passed checks against retained HTML, including the 1,084-character first quote on page two. No continuation chunks were required at the requested response limits, and no Fetch task retry occurred. An initial offline validation regex found zero card blocks; its failed version was preserved and corrected against the same saved responses, without another retrieval.
The successful run made no configuration change during its own task, but its existing Fetch configuration allowed private-address resolutions. An earlier run had stopped at the address-resolution guard before receiving page content. Those are different configurations and outcomes. The later success therefore documents that configured route, rather than proving that the earlier guarded route succeeded or that default configuration would behave identically.
For Fetch, source_url binds a record to the requested page and its retained response. The server-reported final redirect URL, cache headers and origin freshness were not exposed. The article keeps those separate from the known requested URL and client receipt time.
Firecrawl: valid JSON preserved content but assigned the wrong page
Two native Firecrawl scrape calls returned HTML and structured records for the two requested URLs through the configured keyless route. The recorded server identity was firecrawl-fastmcp 3.30.2. All twenty quote texts, authors and tag lists agreed with retained cards. However, every structured page-two record used https://quotes.toscrape.com/ as source_url instead of https://quotes.toscrape.com/page/2/. Both pages were retrieved, but the uncorrected dataset failed full acceptance: ten accepted records and ten rejected records.
The correction kept the original native response and normalized-native.json unchanged. It rebound only the ten affected source URLs to their corresponding native request URL, supported by agreement between metadata.sourceURL and metadata.url. Quote text, punctuation, authors and tag order were unchanged. The correction made no network call and used no other scraper’s records. The corrected handoff then passed all twenty four-field source checks. This is a local provenance correction, not a second claim that the original native JSON was correct.
Earlier blocked attempts remain separate outcomes
The retained earlier Fetch attempt returned no source content because of its address-resolution guard. An earlier Firecrawl attempt had a configured server but no callable native tools in its active client, so it also stopped before retrieval. Their zero-record handoffs explicitly marked both pages incomplete. Zero accepted records cannot establish that a source page contains zero cards.
The Firecrawl screenshot’s comparison against Fetch used that earlier blocked Fetch artifact, so it reported no available matching records at that time. The later successful Fetch run was collected separately. The independent browser audit subsequently compared the later output and confirmed twenty matching identities. The later success does not overwrite the earlier blocker or turn its unavailable comparison into a field-accuracy result.
These runs also use different timing boundaries. Fetch records client dispatch and receipt timestamps, Firecrawl records per-call wall time, and the browser reference includes reasoning, reloads, comparison and viewer preparation. Cache age and server-internal retries remain unknown where unreported. The evidence supports the specific completeness and provenance findings above, with no speed ranking.
How should the workflow handle partial content and errors?
A retrieval workflow should distinguish an accepted dataset from a transport response. Authentication errors, rate limits, network failures and content gaps need different follow-up actions.
For Fetch, inspect truncation before concluding that fields are absent. For a rendered site, confirm whether the chosen server actually returned the populated content. A login page, challenge page or navigation shell cannot satisfy the extraction contract merely because it has text.
Use an explicit retry budget and keep the first failure. Retry only when the observed condition supports it; changing providers or adding another tool changes the route and needs a separate result. Stop at unresolved access restrictions rather than presenting a bypass as recovery.
A controlled negative test can request a deliberately nonexistent quote identity from already retrieved content. The correct response is no matching record. Inventing an author or silently borrowing another run's data is a verification failure.
Treat source-page instructions as untrusted content. The task author supplies the allowed URLs and fields; a page cannot authorize the agent to inspect local files or send credentials elsewhere.
What should the next AI agent receive?
A downstream agent needs the accepted records and the limits of the collection. Pass a small manifest with the run ID, requested pages, pages successfully retrieved, validation status, missing fields and references to the saved source responses. Keep rejected or unresolved records available for review rather than mixing them into the accepted set.
Separate page coverage from record accuracy. If page one is verified and page two is blocked, the workflow can describe a verified subset when the consumer explicitly permits partial data. It must still report incomplete overall coverage. If the task requires all twenty records, hold the handoff until that acceptance condition is met; an empty or partial records array is not a complete result.
Give the consuming agent the verified subset, its source URLs and a clear instruction to cite those sources. Require it to preserve missing values, avoid extending claims beyond the retrieved evidence and request another retrieval when the task needs a new date or an uncollected page. Retrieval content remains evidence to interpret, not instructions that override the user.
For a repeatable workflow, save the normalization and validation code with the raw responses and the final manifest. Check the delivered data after any projection, summarization or field selection. A correctly validated internal object does not help if the next agent receives a shortened field, drops source URLs or loses the page-level completeness status.
The saved handoff audit checked the actual later Fetch payload and corrected Firecrawl payload, alongside their viewers. Each retained all twenty records and all four full fields, with coverage and validation status visible. The Firecrawl handoff also retained the ten native rejected records and a correction reference. Earlier blocked handoffs kept their incomplete pages explicit. No additional field drop or hidden incomplete page was found in those inspected artifacts; this check did not execute a separate downstream AI agent.
Where does ego(lite) help review an extraction?
A visible ego(lite) Space is useful when the returned record needs comparison with the actual page. The ego-browser documentation describes the connection, and the Space guide explains the workspace a person can inspect and take over.
The independent run used the named Space MCP scraping source verification: ego. Source browsing and DOM extraction went through the installed ego-browser helper. It collected twenty unique text-and-author pairs, reloaded both pages and checked every record across all four fields, including the 1,084-character page-two quote. The exact impossible quote had zero matches on both pages and produced no fabricated record.
The independent records were sealed before opening MCP comparison artifacts. Later comparisons matched the latest Fetch and corrected Firecrawl outputs by text-and-author identity across text, author, tags and source_url. The browser source also confirmed the ten incorrect page-two source URLs in native Firecrawl JSON. A separate read-only comparison of the retained files for this article reproduced those exact findings, without repairing any original.
The helper reported ego 0.6.0.5, Chromium 154.0.8037.44 and skill guide 2.1.0. Source collection used a 900 × 750 CSS-pixel viewport at DPR 2; the result viewer was later observed at 888 × 796. One browser-version diagnostic was unsupported and retained. Screenshot capture needed recovery, while no source-retrieval retry or human intervention was observed. No upgrade was performed during the run.
This is a review and collaboration route. It does not automatically share a hosted scraper's session or certify a server's freshness. Recurring bulk collection still needs an explicit retrieval contract and storage outside the visible review session.
FAQ
These questions cover the distinction between connecting a tool and accepting the data it returns.
Does MCP itself scrape websites?
MCP defines the client-server interface. A connected server performs the fetch, extraction or browser interaction through its own implementation.
Is a search snippet enough to feed an agent?
A snippet can help select a source. A task requiring full fields needs the chosen page retrieved and validated separately.
Does every scraping MCP require an API key?
Authentication depends on the server and operation. Firecrawl currently documents limited keyless access; other tools or higher limits can require an account or credential.
Does the reference Fetch server render JavaScript?
The documented Fetch route retrieves content and simplifies HTML. Do not assume it executes the site's browser application; verify the fields actually returned.
Does valid JSON prove extraction accuracy?
Valid JSON proves that the output can be parsed. Source checks are still needed for completeness, values and field associations.
What if the response stops halfway through the page?
Check the server's continuation or truncation behavior. Read remaining chunks when supported, or label the extraction partial instead of accepting it.
Can I merge outputs from multiple servers?
Outputs can be compared after independent collection. Preserve each source result and resolve disagreements against the page before publishing a combined dataset.
How do I know the content is live?
Record retrieval time, exposed metadata and observed page state. If cache age is unavailable, state that limitation rather than claiming a fresh origin fetch.
What should happen when a page blocks access?
Record the actual blocked outcome and the affected fields. Stop or use an authorized access path; don't fill missing records from memory.
Is an ego(lite) result table proof of extraction?
A table is a review aid. Pair it with the real named Space, source page and operation evidence before attributing an extraction to that workflow.
Related reading
The Crawl4AI and Firecrawl guide covers deployment ownership, while JavaScript versus Python scraping compares implementation workflows. For source review that benefits from a visible browser, download ego(lite) and keep the disputed records beside the page.
