Here is what we found out from benchmarking GPT-6.1 Sol against GPT-6 Sol in three browser tools
GPT-6.1 Sol is a clear upgrade for browser agents. We gave it and GPT-6 Sol the same web tasks in three browser tools, and the newer model finished more of them in every one. With Playwright CLI it passed 78.6% of attempts to GPT-6 Sol's 50.0%, with agent-browser 82.1% to 57.1%, and with ego-browser 90.7% to 70.4%. We also found it faster: on the tasks both models passed, it finished 42.9 to 81.9 seconds sooner per task. And it costs less to run, $0.418 per task against $0.787 with Playwright CLI, $0.594 against $0.837 with agent-browser, and $0.422 against $0.857 with ego-browser.
The upgrade took nothing away. We compared GPT-6.1 Sol vs GPT-6 Sol task by task in each tool, 83 task results in all. GPT-6.1 Sol turned 20 of GPT-6 Sol's failures into passes and none of its passes into failures. Another 49 passed with both models, 13 failed with both, and one went from a failure to one pass in two attempts.
When we read through the runs, two changes stood out. GPT-6.1 Sol finishes long tasks inside the 100-step budget: GPT-6 Sol ended 10 of its 83 attempts without a final answer, GPT-6.1 Sol 2 of its 110. It also goes one page further for the exact value a task asks for, where GPT-6 Sol settled for a wrong or partial figure, or marked the value unavailable. Both show up most in multi-site data collection, where 15 of the 20 gains are.
GPT-6.1 Sol is cheaper in all three tools, so there's no extra spend to weigh against the higher pass rate. Part of the saving is OpenAI's lower cached-input price. The rest is usage: on the work both models finished, GPT-6.1 Sol took fewer steps and wrote fewer tokens.
Success rate
GPT-6.1 Sol passed more attempts than GPT-6 Sol in every tool, by 28.6 points with Playwright CLI, 25.0 with agent-browser and 20.3 with ego-browser. That's 22 of 28 against 14 of 28, 23 of 28 against 16 of 28, and 49 of 54 against 19 of 27. The model ranking is the same in all three tools. Task by task, every change went the same way.
GPT-6 Sol failed those 20 tasks in two ways, and GPT-6.1 Sol fixed both. In 7 it ran out of road: six hit the 100-step limit without a final answer, and one ended on an API overload error at step 99. In the other 13 it delivered a full table with a value the task didn't accept. That was a wrong value in 6, an incomplete one in 3, and a value marked unavailable or unverified in 4 that the source does give.
Our open-source security task shows the first kind. It asks for the latest security release of eight projects, matching each project's post on X against its own announcement. With agent-browser, GPT-6 Sol spent about 30 calls on the first project, Go. It ran five versions of its X search there, each followed by a separate call to read the page in slices. It was on Next.js, the fifth project, when it reached step 99, and it never wrote an answer.
GPT-6 Sol reaches step 99 on the fifth of eight projects
export AGENT_BROWSER_SESSION=security-releases-64347d1f960c;agent-browser open https://nextjs.org/blog/security;agent-browser snapshot -i | grep -A2 'September 2026 Security Release';agent-browser get attr 'a:has-text("September 2026 Security Release")' href✓ Next.js Security News | Next.js https://nextjs.org/blog/security- article "September 2026 Security Release" [ref=e5481] - link "Security" [ref=e5498] - heading "September 2026 Security Release" [level=3, ref=e5499] - link "September 2026 Security Release" [ref=e5520]- article "Upcoming Next.js September Security Release" [ref=e5482] - link "Security" [ref=e5500]✓ DoneGPT-6.1 Sol worked through all eight projects in 66 calls. For most projects it submitted the X search, switched to Latest and pulled every post with its timestamp in the same call. It spent its spare steps pinning down the exact time of Go's announcement and rechecking two projects on X. It met every scored item.
GPT-6.1 Sol searches X and reads the results in one call
export AGENT_BROWSER_SESSION=task-64347d1f960c;agent-browser find role combobox fill --name 'Search query' 'from:golang security until:2026-10-01';agent-browser press Enter;agent-browser wait --fn 'document.querySelectorAll("article").length>0 && document.body.innerText.includes("security")';agent-browser read…Go@golang·Mar 6🌟 Go 1.26.1 and 1.25.8 are released!🔐 Security: Includes security fixes for the standard library (CVE-2026-25679, CVE-2026-27137, CVE-2026-27138, CVE-2026-27139, CVE-2026-27142).🗣 Announcement: https://groups.google.com/g/golang-announce/c/EdhZqrQ98hk/m/41DopX_WAAAJ……The value misses are subtler. On the phone task, GPT-6 Sol gave Motorola's security-update promise as seven years with both Playwright CLI and agent-browser. With Playwright CLI, GPT-6.1 Sol read the spec sheet's "6+ years (until 06/2032)" and gave the end date the task asked for. It passed with agent-browser too.
GPT-6.1 Sol still failed 16 of its 110 attempts, and half of those were the site, not the model. In 8, the site served an access-denied, error or region-unavailable page instead of the data. X answered with HTTP 403 errors in the Playwright CLI runs, so both models failed the three tasks that need X posts there. Two more attempts used the full step budget with agent-browser. The other 6 were wrong or missing values in a finished table. On the SaaS compliance task, for example, it labeled Smartsheet's ISO/IEC 42001 status certified when, at the task's cutoff, it was still in progress.
Editor's note: Across 83 task results GPT-6.1 Sol never lost a task GPT-6 Sol passed, which makes this an easy upgrade. The failures left are mostly sites that won't serve the page and single wrong cells, so keep a review step on anything that feeds a decision.
Speed
GPT-6.1 Sol is faster in all three tools, by 15.2% to 17.8% per task on the tasks both models passed. The gain comes from taking fewer steps: it made 22.6% to 33.9% fewer tool calls on the same work and wrote 26.5% to 31.7% fewer output tokens. It was faster on 10 of the 14 shared tasks with Playwright CLI and 13 of 16 with agent-browser.
Where the time goes depends on the tool. With Playwright CLI and agent-browser most of the saving is model time, reasoning in particular: 53.5 seconds of reasoning per task against 111.2, and 49.3 against 122.2. With ego-browser GPT-6.1 Sol spent slightly longer on the model, 161.1 seconds against 147.6. Its fewer browser steps cut tool time from 133.8 seconds to 77.9.
With ego-browser the lead is narrower task by task. GPT-6.1 Sol was faster on 10 of the 19 tasks both passed, and its median time per task was 185.7 seconds against 211.3.
Editor's note: Saving 43 to 82 seconds on a task is easy to notice when someone is waiting on the agent. The fewer steps matter more than the seconds, though, because they are also what keeps GPT-6.1 Sol inside the step budget on long tasks.
Cost
GPT-6.1 Sol is the cheaper model to run in every tool. Across all tasks it cost 46.9% less per task with Playwright CLI, 29.1% less with agent-browser and 50.7% less with ego-browser. On the tasks both models passed it was cheaper on 14 of 14, 15 of 16 and 19 of 19.
Price and usage both fell. OpenAI's list prices, as reported by MIXED, put both models at $2.00 per million input tokens and $10.00 per million output tokens. Cached input is $0.10 for GPT-6.1 Sol against $0.20 for GPT-6 Sol. On the tasks both models passed, GPT-6.1 Sol's tokens priced at GPT-6 Sol's rates come to $0.442, $0.502 and $0.472 per task. That's against GPT-6 Sol's $0.704, $0.600 and $0.719, so fewer tokens account for most of the saving with Playwright CLI and ego-browser.
Cache reads are where the money goes. The agent harness re-sends the conversation on every step, so each extra step is billed again as cached input. Cache reads were 66.8% to 71.6% of GPT-6 Sol's cost and 40.8% to 54.0% of GPT-6.1 Sol's. With agent-browser across all 28 tasks, GPT-6.1 Sol's tokens would have cost more at GPT-6 Sol's prices, $0.915 against $0.837 per task, and the cheaper cached input is what makes it cheaper there.
Editor's note: We'd budget on cost per task, not on the per-token price. Here both moved the same way, so the upgrade pays for itself before counting the extra tasks it finishes.
Multi-site data collection
If your agent collects the same fields from many official sites into one table, switch to GPT-6.1 Sol. This is where the two models differ most. These 12 tasks gather the same fields from 8 to 16 sites each, and pass only when every field is right.
GPT-6.1 Sol passed 7 of 12 multi-site tasks with Playwright CLI against 3, 8 of 12 with agent-browser against 1, and 17 of 22 attempts with ego-browser against 4 of 11. It also cost less per multi-site task in every tool: $0.636 against $1.105, $0.880 against $1.110, and $0.653 against $1.427.
Our SaaS compliance task shows the deeper reading. It asks for nine collaboration vendors' certifications, including each one's highest FedRAMP authorization. With ego-browser, GPT-6 Sol read Asana's trust center and marked Asana as having no FedRAMP authorization, its only miss on the task. GPT-6.1 Sol followed the trust center's Government link, then went on to the Asana Gov page, which says FedRAMP Moderate authorized. It confirmed that in Asana's press release. It passed both attempts.
GPT-6.1 Sol follows Asana Gov to its FedRAMP press release
ego-browser nodejs <<'EOF'const t=await taskSpace(19),p=t.page('p1');await p.goto('https://asana.com/product/asana-gov');console.log(await p.evaluate(()=>[...document.querySelectorAll('a')].filter(a=>/Read more|Learn more/.test(a.innerText)).map(a=>({text:a.innerText,url:a.href}))));EOF[ { text: 'Read more', url: 'https://asana.com/press/releases/pr/asana-achieves-fedramp-moderate-authorization-for-asana-gov/1ceb1491-c003-4907-8458-1536216a16a9' },{ text: 'Learn more', url: 'https://asana.com/trust' } ]Multi-site tasks are still where GPT-6.1 Sol fails most, so keep checking the table. In our runs most of its misses there were one or two cells, such as one SUV's charging power marked unavailable, or a site that wouldn't serve the page.
Which model to use
Switch to GPT-6.1 Sol whichever of these three browser tools your agent uses. It passed more attempts, finished sooner on the tasks both models passed and cost less per task in each one. We found no type of work where GPT-6 Sol came out ahead.
If your agent runs multi-site data collection, the switch matters most, because GPT-6.1 Sol finishes inside the step budget where GPT-6 Sol often didn't. If it works inside business web apps, both models passed every task, so the reason to switch is cost. GPT-6.1 Sol cost $0.212 per task against $0.370 with Playwright CLI, $0.344 against $0.679 with agent-browser, and $0.198 against $0.303 with ego-browser.
Raising GPT-6 Sol's reasoning effort is no substitute. In a separate ego-browser run at max effort, GPT-6 Sol passed 21 of 27 tasks at $1.217 per task. GPT-6.1 Sol at high effort passed 49 of 54 attempts at $0.422.
To get these results, run GPT-6.1 Sol (gpt-6.1-sol) at high reasoning effort in the pi agent harness, with its bash tool and a skill file for the browser tool. ego-browser comes with ego(lite) and drives it as the agent's browser. Playwright CLI and agent-browser drive Google Chrome.
FAQ
Is GPT-6.1 Sol better than GPT-6 Sol for browser agents?
Yes, in every browser tool we tried. On 28 real web tasks run through the same agent harness, GPT-6.1 Sol passed 78.6% of attempts against 50.0% with Playwright CLI, 82.1% against 57.1% with agent-browser and 90.7% against 70.4% with ego-browser. It turned 20 failures into passes and lost none. The gains came mostly on multi-site data collection. The full setup is in How we tested.
Is GPT-6.1 Sol cheaper than GPT-6 Sol?
Yes. In our 28-task browser benchmark, GPT-6.1 Sol cost $0.418 per task against $0.787 for GPT-6 Sol with Playwright CLI, $0.594 against $0.837 with agent-browser, and $0.422 against $0.857 with ego-browser. OpenAI halved the cached-input price, and on shared work GPT-6.1 Sol also used fewer tokens. The saving was smallest with agent-browser. The full setup is in How we tested.
Is GPT-6.1 Sol faster than GPT-6 Sol?
Yes, by 15.2% to 17.8% per task. On the tasks both models passed in our 28-task benchmark, GPT-6.1 Sol took 350.6 seconds against 425.1 with Playwright CLI, 377.6 against 459.5 with agent-browser and 239.8 against 282.7 with ego-browser. It made 22.6% to 33.9% fewer tool calls. With ego-browser it was faster on only 10 of 19 shared tasks. The full setup is in How we tested.
What changed in GPT-6.1 Sol's browser use compared with GPT-6 Sol?
It finishes long tasks and reads further for exact values. In our 28-task benchmark across Playwright CLI, agent-browser and ego-browser, GPT-6 Sol ended 10 of 83 attempts without a final answer and GPT-6.1 Sol 2 of 110. It also took fewer steps on the same work, which leaves more of the 100-step budget for long tasks. The full setup is in How we tested.
Did GPT-6.1 Sol get worse at anything GPT-6 Sol could do?
No, not on our tasks. Across 83 task results in three browser tools, under the same harness and step budget, no task that passed with GPT-6 Sol failed with GPT-6.1 Sol. Twenty went the other way, from failure to pass, and one went from a failure to one pass in two attempts. The full setup is in How we tested.
What does GPT-6.1 Sol still get wrong in a browser?
Mostly sites that won't serve the page, and single wrong cells. GPT-6.1 Sol failed 16 of 110 attempts in our 28-task benchmark across three browser tools. In 8 the site served an access-denied, error or region-unavailable page. Six were wrong or missing values in a finished table, such as a certification marked certified that was still in progress. Two used the full step budget. The full setup is in How we tested.
How does GPT-6.1 Sol do on multi-site data collection?
Much better than GPT-6 Sol. Our 12 multi-site tasks gather fields from 8 to 16 official sites. GPT-6.1 Sol passed 7 against 3 with Playwright CLI, 8 against 1 with agent-browser and 17 of 22 attempts against 4 of 11 with ego-browser. It also cost less per multi-site task in each tool. It still fails more here than anywhere else. The full setup is in How we tested.
Which browser tool works with GPT-6.1 Sol?
GPT-6.1 Sol beat GPT-6 Sol in all three tools we ran it with: Playwright CLI, agent-browser and ego-browser. It passed 78.6%, 82.1% and 90.7% of attempts on our 28 tasks, through the same agent harness and skill-file setup. Each tool pair ran as its own experiment, so these figures compare the two models within a tool. The full setup is in How we tested.
Should I raise GPT-6 Sol's reasoning effort instead of upgrading?
No. In our ego-browser runs, GPT-6 Sol at max reasoning effort passed 21 of 27 tasks at $1.217 per task. GPT-6.1 Sol at high effort passed 49 of 54 attempts at $0.422 per task, on the same tasks, harness and step budget. The newer model gets more done for about a third of the price. The full setup is in How we tested.
Should I switch my browser agent from GPT-6 Sol to GPT-6.1 Sol?
Yes. On 28 web tasks in three browser tools, GPT-6.1 Sol passed more attempts, finished the shared tasks sooner and cost 29.1% to 50.7% less per task. It lost no task GPT-6 Sol passed. The switch matters most for multi-site data collection and other long runs. For business web apps, where both passed everything, the gain is cost. The full setup is in How we tested.
How we tested
Setup
We ran GPT-6.1 Sol (gpt-6.1-sol) and GPT-6 Sol (gpt-6-sol) at high reasoning effort through the same agent harness, pi, with its bash tool and a skill file for each browser tool. The browser tools were Playwright CLI and agent-browser, which drive Google Chrome, and ego-browser, which drives ego(lite). In each tool, both models used the same tool version, skill file, step budget and answer key, and with ego-browser the same ego(lite) and ego-browser versions. The step budget was 100 per attempt. Each tool pair ran as its own experiment, so compare the models within a tool, not the tools with each other.
Tasks
The 28 tasks cover four types of work. Multi-site data collection (12 tasks) gathers the same fields from 8 to 16 official sites, such as the first 2026 rate decision at twelve central banks. Lookups on one to three sites (9) pull and cross-check figures on well-known sites, such as clinical trial registry entries against drug labels. Open-ended research (3) gives a goal and leaves the path open, such as tracing ten citations in a Wikipedia list. Business web app workflows (4) run in test apps, such as matching February invoices to payments. Each type comes from the task's own group in the dataset. The ego-browser runs cover 27 of the 28 tasks, because the eight-camera spec task wasn't in the task list those runs used.
What counts as a pass
Each task has a list of scored items, 197 across the 28 tasks, including one per task that checks the whole deliverable. Of those, 168 are values fixed in advance from the official sites, 18 come from the test apps' generated data and 11 are checked against the live page. A task passes only when its final answer meets every item, and both models were scored against the same answer key.
How we measured
An attempt is one run of a task. With Playwright CLI and agent-browser each model had one attempt per task. With ego-browser GPT-6.1 Sol had two and GPT-6 Sol one, so success rates count attempts and time and cost per task average a task's attempts. A task counts as passed with GPT-6.1 Sol and ego-browser only when both attempts passed. Speed is wall time from start to final answer, split into model time, reasoning included, and tool time. Speed and cost are compared on the tasks both models passed on every attempt: 14 with Playwright CLI, 16 with agent-browser and 19 with ego-browser. Cost is each attempt's recorded token usage at OpenAI's list prices. To split the cost gap into price and usage, we priced GPT-6.1 Sol's tokens at GPT-6 Sol's prices. Attempts that ended without a final answer count as failures. Without them, GPT-6 Sol passed 51.9%, 80.0% and 73.1% with Playwright CLI, agent-browser and ego-browser, and GPT-6.1 Sol 78.6%, 88.5% and 90.7%.
Other runs
Three more ego-browser runs on the same 27 tasks used the same versions, step budget and answer key. GPT-6.1 Sol at high effort with one attempt per task passed 26 of 27 at $0.437 per task, in line with the main run. At max effort, GPT-6.1 Sol passed 26 of 27 at $0.500 and GPT-6 Sol 21 of 27 at $1.217.
Full results
Every task, with each model's result in each browser tool. Pass and Fail are one attempt; 2/2, 1/2 and 0/2 are GPT-6.1 Sol's two ego-browser attempts.
| Task | Type of work | Playwright CLI, GPT-6.1 Sol | Playwright CLI, GPT-6 Sol | agent-browser, GPT-6.1 Sol | agent-browser, GPT-6 Sol | ego-browser, GPT-6.1 Sol | ego-browser, GPT-6 Sol |
|---|---|---|---|---|---|---|---|
| Booking a policy-compliant trip in a test travel app | Business web app workflows | Pass | Pass | Pass | Pass | 2/2 | Pass |
| Bringing tier-1 service monitors in line with a standard | Business web app workflows | Pass | Pass | Pass | Pass | 2/2 | Pass |
| Matching February invoices to payments and opening disputes | Business web app workflows | Pass | Pass | Pass | Pass | 2/2 | Pass |
| Creating purchase orders from stock levels by supplier | Business web app workflows | Pass | Pass | Pass | Pass | 2/2 | Pass |
| Three MacBook configurations from Apple's US store | Lookups on one to three sites | Pass | Pass | Pass | Pass | 2/2 | Pass |
| Six Wes Anderson films' box office on Box Office Mojo and Wikipedia | Lookups on one to three sites | Pass | Pass | Pass | Pass | 2/2 | Pass |
| KYC screen of four UK companies on Companies House | Lookups on one to three sites | Pass | Pass | Pass | Pass | 2/2 | Pass |
| 2021 atopic dermatitis trials against their drug labels | Lookups on one to three sites | Pass | Fail | Pass | Pass | 2/2 | Pass |
| Three CPI series on FRED checked against BLS | Lookups on one to three sites | Pass | Pass | Pass | Pass | 2/2 | Pass |
| Cheapest nonstop fares to Denver on Google Flights | Lookups on one to three sites | Pass | Fail | Fail | Fail | 2/2 | Pass |
| Six web features' browser support on MDN and Can I use | Lookups on one to three sites | Pass | Pass | Pass | Pass | 2/2 | Pass |
| A week of SpaceX launch posts on X against Wikipedia | Lookups on one to three sites | Fail | Fail | Pass | Pass | 2/2 | Pass |
| A YouTube lecture against its GitHub script and arXiv paper | Lookups on one to three sites | Pass | Fail | Pass | Pass | 2/2 | Pass |
| How nine music services list one album | Multi-site data collection | Fail | Fail | Pass | Fail | 0/2 | Fail |
| Official specs of eight cameras | Multi-site data collection | Pass | Fail | Pass | Pass | Not run | Not run |
| First 2026 rate decision at twelve central banks | Multi-site data collection | Pass | Fail | Fail | Fail | 2/2 | Fail |
| Opening keynotes of nine developer conferences | Multi-site data collection | Pass | Pass | Pass | Fail | 2/2 | Pass |
| US specs of nine electric SUVs | Multi-site data collection | Pass | Fail | Fail | Fail | 1/2 | Fail |
| Approval sheet for nine open-weight models on Hugging Face | Multi-site data collection | Fail | Fail | Pass | Fail | 2/2 | Pass |
| Statutory minimum wage in nine jurisdictions | Multi-site data collection | Pass | Pass | Pass | Fail | 0/2 | Fail |
| Official UK specs of nine flagship phones | Multi-site data collection | Pass | Fail | Pass | Fail | 2/2 | Fail |
| Compliance matrix from nine SaaS trust centers | Multi-site data collection | Fail | Fail | Fail | Fail | 2/2 | Fail |
| Free tiers of nine serverless platforms | Multi-site data collection | Pass | Pass | Fail | Fail | 2/2 | Pass |
| Eight open-source projects' security releases on X and their sites | Multi-site data collection | Fail | Fail | Pass | Fail | 2/2 | Pass |
| Eight consumer-electronics launches on X and in press releases | Multi-site data collection | Fail | Fail | Pass | Fail | 2/2 | Fail |
| Reading costs and scores off the ARC Prize leaderboard chart | Open-ended research | Pass | Pass | Pass | Pass | 2/2 | Pass |
| Counting models over EU compute thresholds in Epoch AI's data | Open-ended research | Pass | Pass | Pass | Pass | 2/2 | Pass |
| Tracing ten citations in Wikipedia's list of large language models | Open-ended research | Pass | Fail | Pass | Pass | 2/2 | Fail |
Update log: first published with one run of each model in each of the three browser tools, with two attempts per task for GPT-6.1 Sol in ego-browser.
