DeepSeek V4.1 Flash's browser use benchmarked against GPT-6.1 Sol on 72 web tasks
GPT-6.1 Sol is the model we'd pick for an agent that works across many sites, and DeepSeek V4.1 Flash is the one we'd pick for workflows inside one web app. Each model ran each of our 72 tasks twice in ego-browser, so each model made 144 attempts. GPT-6.1 Sol passed 121 of its 144 attempts (84.0%), and DeepSeek V4.1 Flash passed 82 (56.9%). On the 29 tasks that both models passed in every attempt, GPT-6.1 Sol took 219.7 seconds and $0.310 per task, and DeepSeek V4.1 Flash took 321.7 seconds and $0.047.
Reading the runs, we saw the two models work in different ways. GPT-6.1 Sol often handles several sites in one command, and DeepSeek V4.1 Flash works on one page at a time and keeps looking. On tasks that collect the same fields from 8 or more sites, DeepSeek V4.1 Flash used all 100 of its steps in 23 attempts. In our test apps, where the work happens inside one web app, it nearly matched GPT-6.1 Sol's pass rate and speed at about a sixth of the cost.
Each model also has a weakness we'd plan around. GPT-6.1 Sol sometimes stops looking too soon: on 3 tasks, it failed both attempts, and DeepSeek V4.1 Flash, which kept looking, passed both. DeepSeek V4.1 Flash gave more wrong answers even though it gave fewer answers: 34 of its 144 attempts ended with a wrong answer, and 23 of GPT-6.1 Sol's did.
Success rate, speed and cost
GPT-6.1 Sol leads on success rate and speed, and DeepSeek V4.1 Flash costs about 15% as much. We compare time and cost on the 29 tasks that both models passed in every attempt, so neither model's numbers include failed work. The rows for all 72 tasks include failed attempts.
| Measure | GPT-6.1 Sol | DeepSeek V4.1 Flash |
|---|---|---|
| Success rate, all 72 tasks | 84.0% (121 of 144 attempts) | 56.9% (82 of 144 attempts) |
| Time per task, 29 tasks both passed every time | 219.7 s | 321.7 s |
| Time per task, all 72 tasks | 321.6 s | 511.4 s |
| Cost per task, 29 tasks both passed every time | $0.310 | $0.047 |
| Cost per task, all 72 tasks | $0.483 | $0.071 |
| Cost per passed attempt, all 72 tasks | $0.575 | $0.124 |
| Tool calls per task, all 72 tasks | 34.6 | 61.6 |
Runs of November 7, 2026, with both models in ego-browser. Each model ran each task twice. Time includes waiting for each model's API.
DeepSeek V4.1 Flash costs less because its prices are lower, not because it does less work. It made 61.6 tool calls per task, and GPT-6.1 Sol made 34.6. Even counted per passed attempt, DeepSeek V4.1 Flash cost $0.124, and GPT-6.1 Sol cost $0.575.
Most of the gap in success rate comes from one kind of work. On tasks that collect the same fields from 8 or more sites, GPT-6.1 Sol passed 51 of 64 attempts, and DeepSeek V4.1 Flash passed 21. On web app workflows, the two models passed 29 and 28 of 30 attempts.
| Kind of work | Tasks | GPT-6.1 Sol | DeepSeek V4.1 Flash |
|---|---|---|---|
| Web app workflows | 15 | 96.7% (29/30) | 93.3% (28/30) |
| Search and compare on a few sites | 8 | 93.8% (15/16) | 81.3% (13/16) |
| Research and synthesis | 12 | 66.7% (16/24) | 62.5% (15/24) |
| Checking X posts against other sources | 5 | 100% (10/10) | 50.0% (5/10) |
| Collecting the same fields from 8 or more sites | 32 | 79.7% (51/64) | 32.8% (21/64) |
| All tasks | 72 | 84.0% (121/144) | 56.9% (82/144) |
The success rate is the share of attempts that passed, out of two attempts per task.
On web app workflows, pass rate barely separated the two models, so cost is what sets them apart there. On tasks across many sites, GPT-6.1 Sol passed far more often.
How GPT-6.1 Sol works
GPT-6.1 Sol often handles several sites in one command. When a task spans many sites, it keeps them open in separate tabs and moves several of them forward in each ego-browser command. Both models had the same ego-browser skill file, which explains how to keep several tabs open when a task needs them. On tasks that collect the same fields from 8 or more sites, 56% of GPT-6.1 Sol's browser commands worked in two or more tabs. DeepSeek V4.1 Flash did this in under 1% of its browser commands.
The earthquake task shows what this looks like. It asks for the time, magnitude and depth of one 2025 earthquake as reported by 9 agencies. In one command, GPT-6.1 Sol read the USGS event page, opened the scientific data on EMSC's event page, opened GFZ's event page and sent a fourth tab to INGV. It passed both attempts, with 40 tool calls in one and 32 in the other. DeepSeek V4.1 Flash worked through the 9 agencies one after another in a single tab, filling in each agency's search form in turn, and it used all 100 steps in both attempts.
GPT-6.1 Sol needed fewer tool calls and finished sooner in our runs. On the 29 tasks that both models passed in every attempt, it made 28.6 tool calls per task, and DeepSeek V4.1 Flash made 43.0. On the 16 of those tasks outside web apps, GPT-6.1 Sol was faster on 14. On web app workflows, it worked in two or more tabs in only 3% of its browser commands, and the two models took about the same time.
GPT-6.1 Sol's weakness is that it sometimes stops looking too soon. On 3 tasks, it failed both attempts, and DeepSeek V4.1 Flash passed both. On the LiveBench task, both models searched the leaderboard for GPT-5.4 at High effort and found only its xHigh row. After 12 tool calls, GPT-6.1 Sol reported that the page didn't show that score. DeepSeek V4.1 Flash kept looking and found it, 75.1, in a model picker further down the same page, in its Insights section.
The frontend frameworks task went the same way. In one attempt, GPT-6.1 Sol searched the Next.js release list for "16.3." and reported the top result, 16.3.2 from August 21. DeepSeek V4.1 Flash paged through the release list, which is mostly canary builds, and found 16.3.8, published on September 30. When GPT-6.1 Sol reports that a value isn't on a site, or takes the top result of a search, we'd check that value ourselves before using the answer.
How DeepSeek V4.1 Flash works
DeepSeek V4.1 Flash works on one page at a time and keeps going until it finds what it's looking for. It made 61.6 tool calls per task, and GPT-6.1 Sol made 34.6. That persistence is how it passed the LiveBench and frontend frameworks tasks that GPT-6.1 Sol failed.
Inside one web app, working one page at a time made little difference. Our 15 web app workflows run in test apps that we filled with fresh data, such as a trip booking app and an expense app. DeepSeek V4.1 Flash passed 28 of its 30 attempts there, and GPT-6.1 Sol passed 29. On the 13 workflows that both models passed in every attempt, DeepSeek V4.1 Flash took 173.0 seconds and $0.028 per task, and GPT-6.1 Sol took 169.3 seconds and $0.170.
Booking a policy-compliant trip is a typical example. The task asks the agent to book round-trip flights within the travel policy, choose aisle seats and book a hotel within a mile of the venue. Both models passed both attempts. DeepSeek V4.1 Flash took 189.2 seconds and $0.033 per task, and GPT-6.1 Sol took 199.4 seconds and $0.253.
Across 8 or more sites, the same way of working uses up the steps. On those 32 tasks, DeepSeek V4.1 Flash passed 21 of 64 attempts, and GPT-6.1 Sol passed 51. It made 82.2 tool calls per task there, twice GPT-6.1 Sol's 41.1. Of the 26 attempts in which it used all 100 steps, 23 were on these tasks.
When DeepSeek V4.1 Flash does give an answer, we'd check its conclusions against its own data and the pages it read. It failed 62 attempts: 34 ended with a wrong answer, and 28 ended without an answer. Of those 28, 26 used all 100 steps, and 1 ended when DeepSeek's API returned server errors. All 23 of GPT-6.1 Sol's failed attempts ended with a wrong answer. In one attempt at the flagship phones task, DeepSeek V4.1 Flash's own table listed the Galaxy S26 at 167 g. The line under the table then named the motorola signature, at 186 g, as the lightest phone.
Which model to use
In ego-browser, use GPT-6.1 Sol for an agent that works across many sites. It passed 84.0% of its attempts and never ran out of steps. On tasks that collect the same fields from 8 or more sites, it passed 51 of 64 attempts, and it finished work across several sites sooner.
Use DeepSeek V4.1 Flash for workflows inside one web app. In our test apps, it passed 28 of 30 attempts, took about the same time as GPT-6.1 Sol and cost about a sixth as much. We wouldn't count on it for jobs across 8 or more sites, where it passed 21 of 64 attempts.
For lookups across a few sites and for research tasks, you're choosing between time and money. DeepSeek V4.1 Flash passed 13 of 16 attempts on lookups and 15 of 24 on research, and GPT-6.1 Sol passed 15 of 16 and 16 of 24. DeepSeek V4.1 Flash cost less and took longer on both.
Whichever model you pick, check the key values in its answer. Both models sometimes gave an answer that looked complete and had a wrong value in it. We ran both models at high reasoning effort in ego-browser, which comes with ego(lite) and controls it as the agent's browser.
FAQ
Is DeepSeek V4.1 Flash as good as GPT-6.1 Sol for browser use?
Inside one web app, nearly; across many sites, no. On web app workflows, DeepSeek V4.1 Flash passed 28 of 30 attempts, and GPT-6.1 Sol passed 29. On tasks that collect the same fields from 8 or more sites, DeepSeek V4.1 Flash passed 21 of 64 attempts, and GPT-6.1 Sol passed 51. Over all 72 tasks, they passed 56.9% and 84.0% of their attempts.
Why is GPT-6.1 Sol faster than DeepSeek V4.1 Flash in a browser?
In our runs, GPT-6.1 Sol needed fewer tool calls, partly because it often handles several sites in one command. On tasks that collect the same fields from 8 or more sites, 56% of its browser commands worked in two or more tabs, and under 1% of DeepSeek V4.1 Flash's did. On the 29 tasks that both models passed in every attempt, GPT-6.1 Sol made 28.6 tool calls and took 219.7 seconds per task, and DeepSeek V4.1 Flash made 43.0 and took 321.7. On web app workflows, they took about the same time.
How much cheaper is DeepSeek V4.1 Flash than GPT-6.1 Sol for browser tasks?
DeepSeek V4.1 Flash costs about 15% of what GPT-6.1 Sol costs. On the 29 tasks that both models passed in every attempt, DeepSeek V4.1 Flash cost $0.047 per task, and GPT-6.1 Sol cost $0.310. Across all 72 tasks, DeepSeek V4.1 Flash cost $0.071 per task, and GPT-6.1 Sol cost $0.483. Per passed attempt, DeepSeek V4.1 Flash cost $0.124, and GPT-6.1 Sol cost $0.575.
Why does DeepSeek V4.1 Flash fail tasks across many sites?
About half of its failures there come from using up its steps. On tasks that collect the same fields from 8 or more sites, it works on one page at a time and made 82.2 tool calls per task, against 41.1 for GPT-6.1 Sol. Of its 43 failed attempts on these tasks, 23 ended without an answer after all 100 steps.
Why does GPT-6.1 Sol get some browser tasks wrong?
On LiveBench and on the frontend frameworks task, it stopped looking too soon. On the LiveBench task, after 12 tool calls GPT-6.1 Sol reported that the page didn't show a score, and DeepSeek V4.1 Flash found that score further down the same page, in its Insights section. DeepSeek V4.1 Flash passed both attempts at 3 tasks that GPT-6.1 Sol failed twice.
Does DeepSeek V4.1 Flash make more mistakes than GPT-6.1 Sol?
Yes, 34 of DeepSeek V4.1 Flash's 144 attempts ended with a wrong answer, and 23 of GPT-6.1 Sol's did. DeepSeek V4.1 Flash also gave fewer answers, because 28 of its attempts ended without one. In one attempt at the flagship phones task, its own table listed the Galaxy S26 at 167 g, and its summary named a 186 g phone as the lightest.
Can DeepSeek V4.1 Flash fill forms and run workflows in a web app?
Yes, DeepSeek V4.1 Flash passed 28 of 30 attempts on 15 web app workflows in our test apps, and GPT-6.1 Sol passed 29 of 30. The tasks included booking a trip, applying lockbox payments and approving expense reports. On the 13 tasks that both models passed in every attempt, DeepSeek V4.1 Flash cost $0.028 per task, and GPT-6.1 Sol cost $0.170.
Which browser tasks can DeepSeek V4.1 Flash do for less than GPT-6.1 Sol?
Mostly workflows inside one web app, where 11 of the 15 web app workflows met three conditions. On those 11 tasks, DeepSeek V4.1 Flash passed as often as GPT-6.1 Sol, took at most 10% longer, and cost less than a third as much. Three other tasks met the same conditions. They were an inflation briefing from FRED and BLS data, study notes from a lecture, code and a paper, and one sudo CVE checked across nine Linux distribution advisories.
How many tool calls does DeepSeek V4.1 Flash make compared with GPT-6.1 Sol?
Across all 72 tasks, DeepSeek V4.1 Flash made 61.6 tool calls per task, which is about 78% more than the 34.6 that GPT-6.1 Sol made. On tasks that collect the same fields from 8 or more sites, DeepSeek V4.1 Flash made 82.2 per task, twice the 41.1 that GPT-6.1 Sol made. On web app workflows, DeepSeek V4.1 Flash made 33.3 per task, and GPT-6.1 Sol made 26.0.
Is DeepSeek V4.1 Flash worth it for browser agents?
Yes for workflows inside one web app: DeepSeek V4.1 Flash passed 28 of 30 attempts there at about a sixth of GPT-6.1 Sol's cost. For an agent that works across many sites, GPT-6.1 Sol is the better choice, because it passed 84.0% of its attempts and DeepSeek V4.1 Flash passed 56.9%.
How we tested
We ran both models on November 7, 2026, on macOS, in ego-browser with ego(lite) 0.6.3.2. Both models ran in the pi agent harness with ego-browser's skill file. We called them through the model IDs deepseek-flash and gpt-6.1-sol at high reasoning effort. Both models had the same limit of 100 steps per attempt. A step is one turn of the model, and in almost every step the model made one tool call. Each model ran each of the 72 tasks twice, so each model made 144 attempts.
The tasks are 72 complex web automation tasks. Some tasks run on live websites, and some run in test apps that we filled with fresh data. We grouped the tasks by the kind of work that each task asks for:
| Kind of work | Tasks | Example task |
|---|---|---|
| Web app workflows | 15 | Book a policy-compliant trip |
| Search and compare on a few sites | 8 | Inflation briefing from FRED, checked against BLS |
| Research and synthesis | 12 | Trace a Wikipedia table to its sources |
| Checking X posts against other sources | 5 | SpaceX launch posts on X against Wikipedia |
| Collecting the same fields from 8 or more sites | 32 | Rate decisions at 12 central banks |
Each task is in one group only. The 5 tasks that check X posts form their own group, even when they cover 8 accounts.
Each task is scored on 3 to 10 checks. An attempt passes only when every check passes.
To see how each model works, we read the session logs of both models' attempts. We count a browser command as working in two or more tabs when the script that the model passed to ego-browser in that command read from or acted on two or more tabs.
The success rate is the share of attempts that passed. Every per-task number in this article, including time, cost and tool calls, is the average of the task's two attempts. Time runs from the start of an attempt to its final answer, or to its last step when it gave no answer, and it includes the time spent waiting for each model's API. Cost is the price of each attempt's tokens at the provider's prices. Cost per passed attempt is the cost of all 144 attempts divided by the number that passed. When we compare time and cost, we use the 29 tasks that both models passed in every attempt, unless the sentence says otherwise.
In the Result column below, "Matched for less" means that DeepSeek V4.1 Flash passed as often as GPT-6.1 Sol, took at most 10% longer, and cost less than a third as much. "Matched, slower" means that the two models passed equally often, but the task did not meet those time and cost conditions.
Full results
Each cell shows the number of passed attempts out of 2, then the time per task and the cost per task. Time and cost are averages over the task's two attempts, and time includes waiting for each model's API.
| Task | Kind of work | Result | GPT-6.1 Sol | DeepSeek V4.1 Flash |
|---|---|---|---|---|
| Add unmatched card charges to an expense report | Web app workflows | Matched for less | 2/2, 169.6 s, $0.193 | 2/2, 157.1 s, $0.026 |
| Apply lockbox payments | Web app workflows | Matched for less | 2/2, 185.2 s, $0.141 | 2/2, 184.3 s, $0.025 |
| Approve a travel expense report | Web app workflows | Matched for less | 2/2, 156.3 s, $0.120 | 2/2, 144.6 s, $0.020 |
| Book a policy-compliant trip | Web app workflows | Matched for less | 2/2, 199.4 s, $0.253 | 2/2, 189.2 s, $0.033 |
| Bring service monitors in line with a standard | Web app workflows | Matched for less | 2/2, 133.9 s, $0.131 | 2/2, 119.3 s, $0.024 |
| Carrier on-time review and alert rules | Web app workflows | Matched for less | 2/2, 198.1 s, $0.140 | 2/2, 213.7 s, $0.042 |
| Claim daily meal allowances | Web app workflows | Matched for less | 2/2, 123.4 s, $0.109 | 2/2, 93.2 s, $0.018 |
| Claim daily mileage | Web app workflows | Matched, slower | 2/2, 246.6 s, $0.228 | 2/2, 282.4 s, $0.036 |
| Cover on-call shifts around time off | Web app workflows | Matched for less | 2/2, 85.6 s, $0.108 | 2/2, 69.0 s, $0.013 |
| Create purchase orders from reorder points | Web app workflows | Matched, slower | 2/2, 116.7 s, $0.123 | 2/2, 146.9 s, $0.019 |
| File returns for damaged orders | Web app workflows | Matched, slower | 2/2, 238.5 s, $0.293 | 2/2, 297.5 s, $0.045 |
| Freeze or unfreeze deploys from error budgets | Web app workflows | Matched for less | 1/2, 90.1 s, $0.086 | 1/2, 74.1 s, $0.013 |
| Match invoices to payments, open disputes | Web app workflows | Matched for less | 2/2, 212.0 s, $0.214 | 2/2, 224.6 s, $0.042 |
| Roll back releases made during a freeze | Web app workflows | GPT-6.1 Sol passed more | 2/2, 115.4 s, $0.111 | 1/2, 241.3 s, $0.040 |
| Write and submit an incident postmortem | Web app workflows | Matched for less | 2/2, 135.5 s, $0.158 | 2/2, 126.7 s, $0.026 |
| Cheapest nonstop fares on a flight search site | Search and compare on a few sites | Matched, slower | 1/2, 380.6 s, $0.492 | 1/2, 1058.9 s, $0.096 |
| Compare three Mac laptops | Search and compare on a few sites | Matched, slower | 2/2, 278.5 s, $0.529 | 2/2, 320.4 s, $0.065 |
| Inflation briefing from FRED, checked against BLS | Search and compare on a few sites | Matched for less | 2/2, 117.8 s, $0.158 | 2/2, 93.6 s, $0.019 |
| KYC snapshot of four UK companies | Search and compare on a few sites | GPT-6.1 Sol passed more | 2/2, 181.1 s, $0.269 | 1/2, 170.1 s, $0.034 |
| Match trials to drug labels | Search and compare on a few sites | GPT-6.1 Sol passed more | 2/2, 338.8 s, $0.487 | 1/2, 293.6 s, $0.052 |
| Reconcile six films' box office | Search and compare on a few sites | Matched, slower | 2/2, 189.2 s, $0.209 | 2/2, 210.5 s, $0.028 |
| Study notes from a lecture, code and paper | Search and compare on a few sites | Matched for less | 2/2, 215.6 s, $0.422 | 2/2, 232.5 s, $0.041 |
| Web features on MDN and Can I use | Search and compare on a few sites | Matched, slower | 2/2, 161.8 s, $0.197 | 2/2, 430.5 s, $0.049 |
| API prices from Wayback captures | Research and synthesis | DeepSeek V4.1 Flash passed more | 0/2, 410.4 s, $0.300 | 2/2, 1111.2 s, $0.065 |
| ARC Prize leaderboard chart | Research and synthesis | Matched, slower | 2/2, 197.0 s, $0.247 | 2/2, 341.1 s, $0.081 |
| Check 10 claims in a weekly AI recap | Research and synthesis | GPT-6.1 Sol passed more | 2/2, 184.3 s, $0.331 | 0/2, 307.3 s, $0.059 |
| Compare five Chinese model APIs for a monthly bill | Research and synthesis | Matched, slower | 2/2, 205.8 s, $0.344 | 2/2, 278.6 s, $0.055 |
| EU AI Act disclosure audit of 8 frontier models | Research and synthesis | Neither passed | 0/2, 555.1 s, $1.007 | 0/2, 530.4 s, $0.126 |
| List September API model launches from six changelogs | Research and synthesis | GPT-6.1 Sol passed more | 2/2, 148.2 s, $0.260 | 1/2, 378.5 s, $0.078 |
| Models over EU compute thresholds | Research and synthesis | GPT-6.1 Sol passed more | 2/2, 94.0 s, $0.095 | 1/2, 170.2 s, $0.034 |
| Pick OpenRouter endpoints for GLM 5.3 | Research and synthesis | Matched, slower | 2/2, 222.2 s, $0.305 | 2/2, 363.4 s, $0.067 |
| Pick a model for German insurance claims under six requirements | Research and synthesis | DeepSeek V4.1 Flash passed more | 1/2, 155.3 s, $0.261 | 2/2, 285.0 s, $0.057 |
| Price a support-ticket workload on 6 price pages | Research and synthesis | Matched, slower | 1/2, 334.8 s, $0.559 | 1/2, 399.2 s, $0.101 |
| Read one LiveBench release | Research and synthesis | DeepSeek V4.1 Flash passed more | 0/2, 67.8 s, $0.129 | 2/2, 215.3 s, $0.050 |
| Trace a Wikipedia table to its sources | Research and synthesis | GPT-6.1 Sol passed more | 2/2, 286.6 s, $0.222 | 0/2, 514.5 s, $0.060 |
| 8 club signings, X against club sites | Checking X posts against other sources | GPT-6.1 Sol passed more | 2/2, 1099.5 s, $1.485 | 0/2, 2198.8 s, $0.080 |
| 8 product launches, X posts against press releases | Checking X posts against other sources | GPT-6.1 Sol passed more | 2/2, 583.1 s, $0.725 | 0/2, 1451.6 s, $0.086 |
| 8 universities' Nobel posts on X | Checking X posts against other sources | GPT-6.1 Sol passed more | 2/2, 790.7 s, $0.835 | 1/2, 2557.1 s, $0.117 |
| Security releases of 8 open-source projects, X against advisories | Checking X posts against other sources | Matched, slower | 2/2, 426.5 s, $0.724 | 2/2, 1306.2 s, $0.117 |
| SpaceX launch posts on X against Wikipedia | Checking X posts against other sources | Matched, slower | 2/2, 156.2 s, $0.166 | 2/2, 841.3 s, $0.044 |
| 2025 climate statements from 11 weather agencies | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 1/2, 616.5 s, $1.076 | 0/2, 556.2 s, $0.120 |
| 2025 traffic at 8 airports | Collecting the same fields from 8 or more sites | Matched, slower | 1/2, 609.4 s, $1.015 | 1/2, 750.7 s, $0.127 |
| 2025-26 champions of 9 leagues | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 2/2, 577.7 s, $1.193 | 0/2, 798.0 s, $0.110 |
| 8 cameras from 8 brand sites | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 2/2, 381.1 s, $0.511 | 1/2, 540.1 s, $0.083 |
| 8 developer surveys | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 2/2, 348.6 s, $0.585 | 0/2, 537.0 s, $0.122 |
| 9 electric SUVs | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 2/2, 557.3 s, $1.111 | 0/2, 620.2 s, $0.104 |
| 9 open-weight models | Collecting the same fields from 8 or more sites | Matched, slower | 1/2, 448.1 s, $0.703 | 1/2, 548.0 s, $0.110 |
| Compliance of 9 SaaS vendors | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 1/2, 434.7 s, $0.598 | 0/2, 452.0 s, $0.081 |
| Current editions of 9 standards | Collecting the same fields from 8 or more sites | Matched, slower | 2/2, 282.9 s, $0.675 | 2/2, 386.3 s, $0.059 |
| Flagship phones from 9 brands | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 2/2, 439.4 s, $0.620 | 0/2, 509.0 s, $0.077 |
| Free tiers of 9 CI services | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 2/2, 350.6 s, $0.525 | 0/2, 617.3 s, $0.118 |
| Free tiers of 9 serverless platforms | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 2/2, 209.9 s, $0.319 | 1/2, 390.7 s, $0.077 |
| Keynotes of 9 developer conferences | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 2/2, 429.1 s, $0.586 | 0/2, 949.7 s, $0.114 |
| Latest election in 9 places | Collecting the same fields from 8 or more sites | Neither passed | 0/2, 582.7 s, $1.114 | 0/2, 625.3 s, $0.120 |
| Latest release of 9 language runtimes | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 2/2, 326.9 s, $0.638 | 0/2, 464.1 s, $0.090 |
| Latest stable release of 9 desktop browsers | Collecting the same fields from 8 or more sites | Matched, slower | 2/2, 263.2 s, $0.510 | 2/2, 552.6 s, $0.104 |
| Latest stable versions of 9 frontend frameworks | Collecting the same fields from 8 or more sites | DeepSeek V4.1 Flash passed more | 0/2, 252.0 s, $0.375 | 2/2, 425.9 s, $0.069 |
| Minimum wage in 9 jurisdictions | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 2/2, 520.4 s, $0.834 | 0/2, 630.3 s, $0.088 |
| One album across 9 music services | Collecting the same fields from 8 or more sites | Matched, slower | 2/2, 260.6 s, $0.660 | 2/2, 485.6 s, $0.055 |
| One book's editions across 10 sources | Collecting the same fields from 8 or more sites | DeepSeek V4.1 Flash passed more | 1/2, 315.6 s, $0.477 | 2/2, 498.5 s, $0.061 |
| One car across 9 crash-test programs | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 1/2, 345.1 s, $0.524 | 0/2, 609.2 s, $0.104 |
| One drug across 8 regulators | Collecting the same fields from 8 or more sites | Neither passed | 0/2, 52.7 s, $0.060 | 0/2, 614.5 s, $0.117 |
| One earthquake across 9 agencies | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 2/2, 321.3 s, $0.750 | 0/2, 694.1 s, $0.109 |
| One game across 8 rating boards | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 2/2, 496.2 s, $0.659 | 0/2, 666.1 s, $0.092 |
| One print across 10 museum collections | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 2/2, 465.4 s, $0.777 | 0/2, 693.7 s, $0.098 |
| One sudo CVE across 9 distro advisories | Collecting the same fields from 8 or more sites | Matched for less | 2/2, 389.0 s, $0.487 | 2/2, 280.1 s, $0.045 |
| One week's number one on 9 national charts | Collecting the same fields from 8 or more sites | Matched, slower | 2/2, 530.8 s, $0.771 | 2/2, 631.1 s, $0.090 |
| Q2 GDP first estimates from 9 agencies | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 2/2, 471.3 s, $0.927 | 1/2, 616.9 s, $0.105 |
| Rate decisions at 12 central banks | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 2/2, 374.9 s, $0.620 | 0/2, 496.5 s, $0.109 |
| Support dates for 9 vendor versions | Collecting the same fields from 8 or more sites | Matched, slower | 2/2, 273.8 s, $0.390 | 2/2, 327.4 s, $0.065 |
| The protobuf runtime across 9 package registries | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 1/2, 415.8 s, $0.544 | 0/2, 566.7 s, $0.082 |
| Top prize at 9 film festivals | Collecting the same fields from 8 or more sites | GPT-6.1 Sol passed more | 2/2, 628.3 s, $0.969 | 0/2, 665.1 s, $0.093 |
Update log: November 9, 2026, first published with the runs of November 7, 2026, two attempts per task for each model on 72 tasks.
