We benchmarked Kimi K3.1 Pro against Kimi K3 on 72 browser tasks
Kimi K3.1 Pro is the better browser-agent model of the two, and switching to it costs nothing extra. It passed 36 of 72 attempts (50.0%), against 27 of 72 (37.5%) for Kimi K3. It also cost $0.649 per task, against $0.731. On the 25 tasks both models passed, it took 410.0 seconds per task on average, against 599.5 seconds.
The biggest Kimi K3.1 Pro vs Kimi K3 difference is that the new model finishes what it starts. Kimi K3 terminated 29 of its 72 runs partway through the task, with no final answer. Kimi K3.1 Pro never did: every run that didn't use all 100 steps ended with a full answer. Task by task, it passed 11 that Kimi K3 failed and failed 2 that Kimi K3 passed, and the other 59 came out the same.
Kimi K3.1 Pro is the cheaper model to run. Moonshot lists it at two-thirds of Kimi K3's price on every kind of token. On the work both models finished, it also used fewer tokens.
By type of work, there's none where we'd stay on Kimi K3. Use Kimi K3.1 Pro for web apps automation and search and compare tasks, where it passed every task. On multi-site data collection it beats Kimi K3, but most of its tables there still had a wrong or missing cell. On research and synthesis both passed 5 of 12, and Kimi K3.1 Pro was faster and cheaper on the 4 both passed.
Success rate
Kimi K3.1 Pro passed 50.0% of attempts (36 of 72) and Kimi K3 37.5% (27 of 72). Counted by individual scored items, it met 320 of 489 against 199. It came out ahead in four of the five types of work and lost no task in web apps automation, search and compare or social media verification. Research and synthesis is the exception, level at 5 of 12 with one gain and one loss.
Kimi K3.1 Pro passed more tasks in four of five types of work
What Kimi K3.1 Pro fixed is premature termination. In 29 of its 45 failed tasks, Kimi K3 returned a text-only response with no tool call partway through the task, which ends the agent loop before a final answer. Nineteen of those 29 were multi-site data collection, where the agent visits eight or more sites in sequence. Among the 11 tasks only Kimi K3.1 Pro passed, Kimi K3 had terminated this way in 7 and answered with a wrong or missing value in 3, and the eleventh also ended without an answer.
Our serverless task shows the pattern. It asks for five limits on the free plans of nine serverless platforms, from AWS Lambda to Oracle OCI Functions. Kimi K3 read the pages for AWS, Azure and Google, then stopped before the fourth platform, Cloudflare Workers, after 11 tool calls and with no table. Kimi K3.1 Pro continued to Cloudflare's pricing and limits pages, worked through all nine platforms and met all 7 scored items.
Kimi K3.1 Pro reaches Cloudflare Workers, the platform where Kimi K3 stopped
Step 10 of 55 · Cloudflare Workers pricing
ego-browser nodejs <<'EOF'const task=await taskSpace(64),p=task.page('p1');await p.goto('https://developers.cloudflare.com/workers/platform/pricing/',{timeout:60000});console.log({url:await p.url(),title:await p.title()});console.log(await p.evaluate(()=>{const l=document.body.innerText.split('\n');return l.map((s,i)=>({s,i})).filter(x=>/Free|100,000|requests per day|GB.second|memory|CPU/i.test(x.s)).slice(0,90).map(x=>l.slice(Math.max(0,x.i-2),x.i+5).join(' | ')).join('\n---\n')}));EOF{ url: 'https://developers.cloudflare.com/workers/platform/pricing/', title: 'Pricing · Cloudflare Workers docs' }…Free 100,000 per day No charge for duration 10 milliseconds of CPU time per invocation | ……Kimi K3.1 Pro lost two tasks Kimi K3 passed. On an Azure model pick it ruled one candidate out without opening the page that says why, which the research and synthesis section covers below. On the national singles charts, the UK chart's site returned an error page throughout Kimi K3.1 Pro's attempt, so it marked the UK row not shown and missed every scored item. Kimi K3's attempt loaded that page.
However, 36 tasks still failed with Kimi K3.1 Pro, 26 of them in multi-site data collection. In 4 it used all 100 steps without an answer. The other 32 ended with a full answer the answer key didn't accept. It had a wrong value in 18 and a value marked not shown that the source gives in 11. In the last 3, figures from one site had changed since the answer key was fixed or came back as an error page. On the statutory minimum wage task, it reported Tokyo's ¥1,226 an hour as the rate on 30 September 2026, in force since 3 October 2025. It then gave the same ¥1,226 for a year earlier, before that rate began, although the Ministry of Health, Labour and Welfare spreadsheet it had downloaded lists ¥1,163. That made Tokyo's rise 0.0% and its summary named Tokyo, not New Zealand, as the smallest rise.
Speed
Kimi K3.1 Pro is faster. On the 25 tasks both models passed, it averaged 410.0 seconds per task against 599.5 (medians 387.7 and 535.9), and it was faster on 20 of the 25.
Average time per task, lower is better
Most of the gap is reasoning. Kimi K3.1 Pro spent 78.1 seconds per task reasoning before its steps, against 242.6 for Kimi K3, and produced 2,130 reasoning tokens against 4,404. It also needed fewer steps, 26.3 tool calls per task against 34.4, so the time spent running them fell from 110.3 seconds to 94.1. Kimi K3 was faster on 5 tasks: two search and compare tasks, two web apps automation tasks and the one social media verification task both models passed.
Most of the time saved is reasoning
Cost
Kimi K3.1 Pro costs less. It cost $0.649 per task across all 72 tasks against $0.731 for Kimi K3, 11% less. On the 25 tasks both models passed, it cost $0.273 against $0.473 and was cheaper on 23 of them. Divided by the tasks each model passed, that's $1.299 per passed task against $1.950.
Price does most of the work. Moonshot lists Kimi K3.1 Pro at $2.00 per million input tokens, $0.20 cached and $10.00 output, two-thirds of Kimi K3's $3.00, $0.30 and $15.00. On the work both models finished it also used fewer tokens.
Cheaper per task overall and on most types of work
Across all 72 tasks Kimi K3.1 Pro used about 1.5 times as many tokens, mostly cache reads (2.60 million per task against 1.66 million). All of that extra went into the 29 tasks where Kimi K3 stopped partway, which Kimi K3.1 Pro kept working on. Cache reads are most of both bills, 80.2% of Kimi K3.1 Pro's spend and 68.2% of Kimi K3's, because the agent harness re-sends the conversation on every step. Moonshot also lists a cache-write price, which these figures don't include. Priced at its lower write rate, the same as input, cache writes would bring the totals to $0.831 against $0.891, and Kimi K3.1 Pro stays the cheaper model.
By type of work, Kimi K3.1 Pro was cheaper on web apps automation ($0.168 against $0.305), search and compare ($0.347 against $0.528) and research and synthesis ($0.459 against $0.679). It cost about the same on multi-site data collection ($1.034 against $1.037) and more on social media verification ($0.570 against $0.508), where Kimi K3 had terminated partway through 4 of the 5 tasks.
Web apps automation
If your agent automates web apps, switch to Kimi K3.1 Pro. Our 15 tasks run in test apps for operations, travel expenses and a cloud console, which generate fresh data for each attempt. Kimi K3.1 Pro passed all 15 against 13, at $0.168 per task against $0.305. On the 13 both passed it averaged 327.0 seconds against 454.7 and was faster on 11.
The carrier review is one of its two gains. It asks for three carriers' February on-time rates and for the alert rules to match them. Kimi K3 reached the alert-rules page and stopped while updating the rules, without reporting any figures. Kimi K3.1 Pro computed the three rates, replaced Ridgeway Freight's incomplete rule with one that notifies the operations team and the carrier account manager, and reported all three rows.
Kimi K3.1 Pro saves Ridgeway Freight's corrected on-time rule
Step 22 of 22 · Carrier alert rules
ego-browser nodejs <<'EOF'const task = await taskSpace(128);const page = task.page("p1");await page.click("@47", {label:"save corrected Ridgeway alert"});await page.acceptDialog();await page.waitForTimeout(700);console.log(await page.snapshot({scope:"full_page"}));EOF@47 is the form's Save rule button. The same call accepts the confirmation dialog.… table_row table_cell text "ALR-K4XMM" table_cell text "Ridgeway Freight" table_cell text "On-time rate" text "falls below" text "80" table_cell text "Operations team, Carrier account manager"…Multi-site data collection
If your agent runs multi-site data collection, Kimi K3.1 Pro is the better of the two, but don't run it unattended yet. It passed 6 of our 32 multi-site tasks against 3, gaining 4 and losing 1, at about the same cost per task ($1.034 against $1.037). Each task gathers the same fields from eight or more sites into one table and passes only if every field is right.
Our GDP task shows what improved besides finishing. It asks nine statistics agencies for the first and latest Q2 2026 growth figures, not annualized. Kimi K3 read all three of the US Bureau of Economic Analysis's releases, reasoned that BEA prints only annualized rates and marked both US cells not shown. Kimi K3.1 Pro read the same releases further down, where BEA's advance estimate gives "0.4 percent at a quarterly rate" for the first estimate and 0.6 percent for the latest, and filled both cells.
Kimi K3.1 Pro reads BEA's release down to the quarterly rate
Step 2 of 61 · BEA advance estimate
ego-browser nodejs <<'EOF'const task = await taskSpace("eff4-high-gdp-first-estimates-01-…");const page = task.page("p1");await page.goto("https://www.bea.gov/news/2026/gdp-advance-estimate-2nd-quarter-2026", {timeout:30000});console.log({url:await page.url(),title:await page.title()});console.log(await page.evaluate(()=>({text:document.querySelector('main')?.innerText||document.body.innerText})));EOF…Next release: August 26, 2026, at 8:30 a.m. EDTGDP (Second Estimate) and Corporate Profits, 2nd Quarter 2026Technical NotesSources of change for real GDPReal GDP increased at an annual rate of 1.5 percent (0.4 percent at a quarterly rate1) in the second quarter, ……This is still where Kimi K3.1 Pro fails most: 26 of its 36 failures are multi-site tasks, 24 of them a full table with at least one wrong or empty cell. Check the empty and derived cells before you use the table.
Research and synthesis
For research and synthesis, switching from Kimi K3 to Kimi K3.1 Pro buys speed and cost, not more passes. These 12 tasks give the agent a goal and a few sites and leave the path to it. Both models passed 5, with one gain and one loss each. On the 4 both passed, Kimi K3.1 Pro took 402.1 seconds per task against 930.8 and cost $0.313 against $0.883.
Its loss was the Azure Germany model pick. Both models chose GPT-6 Luna as the one candidate that meets all six requirements. Kimi K3 also opened Azure's page for models from partners and quoted why Claude Sonnet 5.5 doesn't qualify: its Data Zone Standard deployment is US-only. Kimi K3.1 Pro ruled Claude out only because it was missing from the table of models Azure sells directly, so it had no value to give for that verdict and cited the wrong page. With either model we'd ask for the evidence behind every option it rules out.
Which model to use
If your agent automates web apps or searches and compares figures across a few sites, switch to Kimi K3.1 Pro. It passed every one of those tasks, faster and for less per task. If your agent runs multi-site data collection into one table, switch too, since Kimi K3.1 Pro finishes the table where Kimi K3 often stopped partway. Keep a review step on that table, because most of them still had a wrong or missing cell.
Kimi K3 has no type of work where it comes out ahead. It tied on research and synthesis, passed fewer tasks everywhere else and cost less only on social media verification, where it also passed fewer.
To get these results, run Kimi K3.1 Pro (kimi-k3.1-pro) at high reasoning effort in the pi agent harness, with its bash tool and a skill file for ego-browser. ego-browser comes with ego(lite) and drives it as the agent's browser.
FAQ
Is Kimi K3.1 Pro better than Kimi K3 for browser agents?
Yes, clearly. Kimi K3.1 Pro passed 50.0% of attempts (36 of 72) against 37.5% (27 of 72) for Kimi K3. Both ran the same 72 web tasks in ego-browser under the same agent harness and step budget. The gain comes from finishing: Kimi K3 terminated 29 runs partway through with no answer, and Kimi K3.1 Pro never did. Research and synthesis didn't improve, level at 5 of 12. The full setup is in How we tested.
Is Kimi K3.1 Pro cheaper than Kimi K3?
Yes. In our 72-task browser benchmark, Kimi K3.1 Pro cost $0.649 per task and Kimi K3 $0.731. On the 25 tasks both passed, it was $0.273 against $0.473. Moonshot lists it at two-thirds of Kimi K3's price, and on shared work it used fewer tokens too. It cost slightly more only on social media verification, where Kimi K3 often terminated partway through. The full setup is in How we tested.
Is Kimi K3.1 Pro faster than Kimi K3?
Yes. On the 25 tasks both models passed in our 72-task benchmark, Kimi K3.1 Pro took 410.0 seconds per task and Kimi K3 599.5 seconds. It was faster on 20 of the 25. Most of the gap is reasoning time, 78.1 seconds per task against 242.6, plus fewer tool calls, 26.3 against 34.4. Kimi K3 was still faster on 5 of the 25. The full setup is in How we tested.
How much does Kimi K3.1 Pro cost per million tokens?
Moonshot lists Kimi K3.1 Pro at $2.00 per million input tokens, $0.20 cached input and $10.00 output, and cache writes from $2.00. Kimi K3 is listed at $3.00, $0.30 and $15.00. In our 72-task browser benchmark that came to $0.649 per task against $0.731, not counting cache writes. Cache reads were most of both bills. The full setup is in How we tested.
Why does Kimi K3 stop in the middle of a browser task?
In our runs, Kimi K3 sometimes returned a text-only response with no tool call partway through a task, which ends the agent loop before a final answer. It happened on 29 of 72 tasks, most of them multi-site data collection, where the agent visits multiple sites in sequence. Kimi K3.1 Pro, run in the same harness, never terminated this way. The full setup is in How we tested.
Which browser tasks improved the most with Kimi K3.1 Pro?
Search and compare improved most: Kimi K3.1 Pro passed 8 of 8 tasks against 5, in our 72-task benchmark run in ego-browser under the same harness. Social media verification rose from 1 of 5 to 2 of 5, and web apps automation from 13 of 15 to all 15. Multi-site data collection gained the most tasks, 4, but still passed only 6 of 32. The full setup is in How we tested.
Did Kimi K3.1 Pro get worse on any browser task?
On pass or fail, on two of 72. On an Azure model pick, Kimi K3.1 Pro ruled a candidate out without opening the page that says why. On national singles charts, the UK chart's site returned an error page throughout its attempt. Kimi K3.1 Pro passed 11 tasks Kimi K3 failed. By scored items it met fewer than Kimi K3 on 5 tasks, 3 of which both models failed. The full setup is in How we tested.
What does Kimi K3.1 Pro still get wrong in a browser?
Mostly wrong or missing values in multi-site tables. Of its 36 failures in our 72-task benchmark, 26 were multi-site data collection. Its full answers had a wrong value in 18 and a value marked not shown that the source gives in 11. It used all 100 steps on 4 tasks. On one task it gave Tokyo's current minimum wage for the previous year too. The full setup is in How we tested.
How does Kimi K3.1 Pro handle web apps automation?
Kimi K3.1 Pro passed all 15 web apps automation tasks in our benchmark, against 13 for Kimi K3. The tasks run in test apps for operations, travel expenses and a cloud console, with fresh data for each attempt. It cost $0.168 per task against $0.305, and averaged 327.0 seconds against 454.7 on the 13 both passed. The full setup is in How we tested.
Should I switch my browser agent from Kimi K3 to Kimi K3.1 Pro?
Yes. In our 72-task benchmark, Kimi K3.1 Pro passed 36 tasks against 27 for Kimi K3. It cost $0.649 per task against $0.731 and was faster on 20 of the 25 tasks both passed. The upgrade matters most for long, multi-step runs, where Kimi K3 often stopped before answering. For multi-site tables, keep a review step. The full setup is in How we tested.
What settings did you use to test Kimi K3.1 Pro?
We ran kimi-k3.1-pro and kimi-k3 at high reasoning effort through the pi agent harness, with ego-browser as the browser tool. Each model got the same 72 tasks once, with a budget of 100 steps per task. More detail is in How we tested, and every task is in Full results.
How we tested
Setup
We ran Kimi K3.1 Pro and Kimi K3 through the same agent harness, pi, at high reasoning effort, with ego-browser as the browser tool and the same ego-browser skill file. Both runs used the same ego(lite) and ego-browser versions. The model drives ego-browser by writing short JavaScript programs that load pages, click, fill forms and print page text or an accessibility snapshot. Each model ran each of the 72 tasks once, with a budget of 100 steps per task.
Tasks
The 72 tasks cover five types of work. Multi-site data collection (32 tasks) gathers the same fields from eight or more official sites, such as nine statistics agencies' GDP releases. Research and synthesis (12) gives a goal and a few sites, such as picking a model offered in one Azure region. Search and compare (8) tasks compare figures across one to three sites, such as clinical trial registry entries against drug labels. Social media verification (5) matches posts on X against other records, such as SpaceX's launch posts against Wikipedia's launch lists. Web apps automation (15) runs in test apps for operations, travel expenses and a cloud console, such as settling carriers' on-time figures and fixing alert rules. Each type comes from the task's own group in the dataset.
What counts as a pass
Each task has a list of scored items, 489 in all, including one per task that checks the whole deliverable. 404 are values fixed in advance from the official sites, 64 come from the business apps' generated data and 21 are checked against the live page. A task passes only when its final answer meets every item, and both models were scored by the same rules. Items from the business apps were checked against the data generated for each attempt.
How we measured
An attempt is one run of a task, and each model had one attempt per task, so time and cost per task are those of its attempt. Time per task is wall time from start to the end of the run, split into model time, reasoning included, and tool time. Speed is compared on the 25 tasks both models passed, since time spent on a failed task can't be compared. Cost is reported across all 72 tasks and on those 25, from each attempt's recorded token usage at Moonshot's list prices for input, cache reads and output. Attempts that used all 100 steps count as failures: 4 for Kimi K3.1 Pro and 5 for Kimi K3. Without them, the pass rates are 52.9% (36 of 68) and 40.3% (27 of 67).
Full results
Every task, with each model's result, how many of its scored items the final answer met, and its time and cost.
| Task | Type of work | Kimi K3.1 Pro | Kimi K3 | Result |
|---|---|---|---|---|
| 2025 climate scorecard from eleven weather services | Multi-site data collection | Fail1/7 | Fail0/7 | Both failed |
| 2025 traffic at eight hub airports | Multi-site data collection | Fail1/8 | Fail0/8 | Both failed |
| 2025-26 season figures from nine football leagues | Multi-site data collection | Fail0/8 | Fail0/8 | Both failed |
| 2026 top prizes at nine film festivals | Multi-site data collection | Fail5/8 | Fail0/8 | Both failed |
| Approval sheet for nine open-weight models on Hugging Face | Multi-site data collection | Fail4/8 | Fail0/8 | Both failed |
| Compliance matrix from nine SaaS trust centers | Multi-site data collection | Fail4/7 | Fail0/7 | Both failed |
| Crash-test ratings for one SUV across nine programs | Multi-site data collection | Fail4/7 | Fail0/7 | Both failed |
| Editions in force for nine technical standards | Multi-site data collection | Fail2/7 | Fail0/7 | Both failed |
| First 2026 rate decision at twelve central banks | Multi-site data collection | Fail4/7 | Fail5/7 | Both failed |
| First and latest Q2 2026 GDP prints at nine agencies | Multi-site data collection | Pass8/8 | Fail4/8 | Only Kimi K3.1 Pro passed |
| Free CI plans for private repos at nine services | Multi-site data collection | Fail3/7 | Fail0/7 | Both failed |
| Free tiers of nine serverless platforms | Multi-site data collection | Pass7/7 | Fail0/7 | Only Kimi K3.1 Pro passed |
| How nine music services list one album | Multi-site data collection | Fail2/7 | Fail3/7 | Both failed |
| Latest election results from nine election authorities | Multi-site data collection | Fail0/8 | Fail0/8 | Both failed |
| Newest release line of nine language runtimes | Multi-site data collection | Fail4/7 | Fail4/7 | Both failed |
| Newest stable release of nine JavaScript frameworks | Multi-site data collection | Pass6/6 | Pass6/6 | Both passed |
| Newest stable release of nine desktop browsers | Multi-site data collection | Fail2/6 | Fail0/6 | Both failed |
| Nine agencies' readings of the 2025 Kamchatka earthquake | Multi-site data collection | Fail5/7 | Fail0/7 | Both failed |
| Official UK specs of nine flagship phones | Multi-site data collection | Pass7/7 | Fail0/7 | Only Kimi K3.1 Pro passed |
| Official specs of eight cameras | Multi-site data collection | Pass7/7 | Pass7/7 | Both passed |
| One book's edition details on ten publisher and library sites | Multi-site data collection | Fail0/8 | Fail0/8 | Both failed |
| One game's age ratings at eight rating boards | Multi-site data collection | Fail3/7 | Fail0/7 | Both failed |
| One sudo CVE's fix status in nine vendor advisories | Multi-site data collection | Fail1/7 | Fail0/7 | Both failed |
| One week's number-one singles on nine national charts | Multi-site data collection | Fail0/8 | Pass8/8 | Only Kimi K3 passed |
| Opening keynotes of nine developer conferences | Multi-site data collection | Fail1/7 | Fail0/7 | Both failed |
| Protocol Buffers releases on nine package registries | Multi-site data collection | Fail4/8 | Fail0/8 | Both failed |
| Results from eight 2025 developer surveys | Multi-site data collection | Fail4/7 | Fail5/7 | Both failed |
| Statutory minimum wage in nine places | Multi-site data collection | Fail4/7 | Fail0/7 | Both failed |
| Support end dates for nine server products | Multi-site data collection | Pass7/7 | Fail0/7 | Only Kimi K3.1 Pro passed |
| Ten museums' records of Hokusai's Great Wave | Multi-site data collection | Fail6/8 | Fail0/8 | Both failed |
| Tirzepatide weight-management approvals at eight regulators | Multi-site data collection | Fail0/7 | Fail0/7 | Both failed |
| US range and charging figures for nine electric SUVs | Multi-site data collection | Fail5/7 | Fail0/7 | Both failed |
| Cheap reasoning settings on the ARC Prize leaderboards | Research and synthesis | Pass6/6 | Pass6/6 | Both passed |
| Count models above EU compute markers in Epoch AI's data | Research and synthesis | Fail4/9 | Fail4/9 | Both failed |
| EU AI Act disclosures for eight frontier models | Research and synthesis | Fail0/11 | Fail0/11 | Both failed |
| Every model launched in September 2026, from six API changelogs | Research and synthesis | Pass7/7 | Pass7/7 | Both passed |
| Fact-check ten claims about September 2026 model launches | Research and synthesis | Fail1/5 | Fail1/5 | Both failed |
| Pick a main and backup API among five Chinese models | Research and synthesis | Pass6/6 | Pass6/6 | Both passed |
| Pick a model offered in one Azure Germany region | Research and synthesis | Fail4/7 | Pass7/7 | Only Kimi K3 passed |
| Pick the fastest GLM 5.3 provider endpoint on OpenRouter | Research and synthesis | Pass10/10 | Fail9/10 | Only Kimi K3.1 Pro passed |
| Price one API workload across six models | Research and synthesis | Pass9/9 | Pass9/9 | Both passed |
| Six providers' API prices as the Wayback Machine kept them | Research and synthesis | Fail2/9 | Fail0/9 | Both failed |
| Trace the sources behind a Wikipedia LLM table | Research and synthesis | Fail5/8 | Fail0/8 | Both failed |
| What one LiveBench release showed | Research and synthesis | Fail5/7 | Fail5/7 | Both failed |
| Build an inflation table from FRED and cross-check it on BLS | Search and compare | Pass8/8 | Pass8/8 | Both passed |
| Check a GPT lecture's numbers against the video, code and paper | Search and compare | Pass6/6 | Pass6/6 | Both passed |
| Compare three MacBook configurations on Apple's US store | Search and compare | Pass9/9 | Pass9/9 | Both passed |
| Cross-check Phase 3 trials on ClinicalTrials.gov against DailyMed labels | Search and compare | Pass9/9 | Fail0/9 | Only Kimi K3.1 Pro passed |
| Go/no-go list for six web features from MDN and Can I use | Search and compare | Pass5/5 | Pass5/5 | Both passed |
| Price a team offsite's flights to Denver | Search and compare | Pass11/11 | Fail0/11 | Only Kimi K3.1 Pro passed |
| Reconcile six Wes Anderson films' grosses between a box-office database and Wikipedia | Search and compare | Pass9/9 | Pass9/9 | Both passed |
| Register snapshot of four UK companies on Companies House | Search and compare | Pass9/9 | Fail7/9 | Only Kimi K3.1 Pro passed |
| Eight brands' 2026 product announcements on X and newsrooms | Social media verification | Fail3/7 | Fail0/7 | Both failed |
| Eight football clubs' summer signings on X and club sites | Social media verification | Fail0/7 | Fail0/7 | Both failed |
| Eight open-source projects' security releases on X and their sites | Social media verification | Pass7/7 | Fail0/7 | Only Kimi K3.1 Pro passed |
| How fast eight universities posted their 2025 Nobel news on X | Social media verification | Fail5/7 | Fail0/7 | Both failed |
| Match SpaceX's launch posts on X to Wikipedia's launch lists | Social media verification | Pass5/5 | Pass5/5 | Both passed |
| Apply a lockbox deposit to open invoices | Web apps automation | Pass4/4 | Pass4/4 | Both passed |
| Audit and roll back changes made during a change freeze | Web apps automation | Pass4/4 | Pass4/4 | Both passed |
| Book a colleague's trip in a travel app | Web apps automation | Pass5/5 | Pass5/5 | Both passed |
| Bring tier-1 monitors up to a written standard | Web apps automation | Pass4/4 | Fail0/4 | Only Kimi K3.1 Pro passed |
| Claim a week of client-visit mileage | Web apps automation | Pass4/4 | Pass4/4 | Both passed |
| Close February receivables and open disputes in an operations app | Web apps automation | Pass5/5 | Pass5/5 | Both passed |
| File a two-city per diem claim | Web apps automation | Pass4/4 | Pass4/4 | Both passed |
| File an incident postmortem timeline in a cloud console | Web apps automation | Pass4/4 | Pass4/4 | Both passed |
| File damage claims for two customers' deliveries | Web apps automation | Pass4/4 | Pass4/4 | Both passed |
| Fix next week's on-call schedule | Web apps automation | Pass4/4 | Pass4/4 | Both passed |
| Match card transactions to a draft expense report | Web apps automation | Pass4/4 | Pass4/4 | Both passed |
| Place a warehouse's replenishment orders | Web apps automation | Pass4/4 | Pass4/4 | Both passed |
| Review an employee's expense report | Web apps automation | Pass4/4 | Pass4/4 | Both passed |
| Set deploy freezes from error-budget burn | Web apps automation | Pass4/4 | Pass4/4 | Both passed |
| Settle carrier on-time figures and fix alert rules | Web apps automation | Pass6/6 | Fail0/6 | Only Kimi K3.1 Pro passed |
Update log: first published with one run of each model on all 72 tasks.
Model prices are Moonshot's published list prices. All other numbers come from our own runs.
