Patronus AI’s SpeedrunBench asks frontier agents to do what human speedrunners have done for decades: stop worrying about whether you can finish the game and start shaving frames.
On September 3, 2026, Patronus AI released SpeedrunBench, a new benchmark that treats video games less like puzzles to solve and more like clocks to beat.
The question is no longer simply: Can an AI agent complete Pokémon Blue?
It is: How fast can it do it — and can it make the next run faster than the last?
That distinction matters more than it might seem.
Once frontier agents become capable of finishing a game reliably, traditional completion benchmarks start losing their usefulness. If every leading model eventually reaches the credits, a pass/fail scoreboard tells you very little about which one actually plans better.
Human speedrunners figured this out decades ago. They play the same games again and again, but change the objective from completion to optimization. Every unnecessary movement matters. Every bad route costs time. Every missed input is another frame left on the table.
Communities like speedrun.com and TASVideos have spent years discovering shortcuts, route improvements, frame-perfect jumps, and techniques that are completely unnecessary for beating a game — but essential for beating it quickly.
SpeedrunBench brings that same idea to AI evaluation.
The paper, benchmark website, interactive demo, official launch post, and Hugging Face dataset all went live alongside the release.
Games have long been attractive AI testbeds because they give researchers something the real world rarely does: repeatable environments with exact outcomes and exact scores.
Atari, Go, StarCraft, and more recently benchmarks such as LMGame-Bench and VideoGameBench have all used games to test increasingly capable systems.
But many recent LLM gaming benchmarks still revolve around one basic question: Can the model finish?
That metric becomes less useful once several frontier models can.
Speedrunning brings the gradient back.
Imagine two agents that both reach Brock’s gym in Pokémon Blue. One needs 160,000 frames. The other gets there in 40,000.
Technically, both succeeded.
Capability-wise, they are nowhere near each other.
Closing that gap requires more than following instructions. It demands several behaviors that increasingly matter for autonomous AI systems:
Patronus argues that these skills map naturally onto longer agentic tasks such as navigating large codebases, debugging complex systems, or iterating on research.
The analogy is straightforward: a speedrun is a long plan that must constantly be improved without collapsing.
So is a multi-hour coding task.
SpeedrunBench spans ten games across multiple genres and platforms, mixing commercial titles with open-source alternatives.
That mix is deliberate. By pairing familiar games with less widely documented ones, researchers can better distinguish between agents that are actually learning to play and agents that may be reproducing strategies encountered during pretraining.
Agents are then evaluated in three different settings.
This is the most conventional setup.
The agent sees the current game frame and responds with the next button press — or a short sequence of inputs.
It tests live visual understanding, timing, and control.
It also exposes an uncomfortable weakness: a model may be capable of excellent reasoning and still struggle when that reasoning has to happen inside a 60 fps environment.
Here, the agent does not need to react to the game in real time.
Instead, it receives a complete frame-indexed action trace that already reaches the goal. Its job is to modify that trace, replay it, measure the new frame count, and keep iterating within a fixed budget.
This is essentially an autoresearch loop:
Study the existing solution.
Form a hypothesis.
Change the actions.
Run the experiment.
Measure the result.
Try again.
The model is no longer just playing the game. It is studying its own performance.
This uses the same iterative optimization loop — except there is no starting trace.
The agent first has to discover a route that works at all.
Then it has to make that route faster.
Discovery comes first. Optimization comes second.
These offline modes also make the benchmark more broadly useful because smaller open-weight models can compete on the same optimization objective without needing to process every frame of live gameplay.
The early results suggest SpeedrunBench is nowhere near saturated.
And the harder the game becomes, the larger the gap between completing it and optimizing it appears to grow.
On simple stages, frontier agents can already approach elite human performance.
With enough evaluation attempts — up to 200 runs per optimization iteration — GLM 5.2 finished Super Mario Land 1-1 just 58 frames behind the human world record.
That is roughly 0.97 seconds.
On short, highly constrained environments, AI agents are already getting remarkably close.
Longer RPGs expose a much bigger weakness.
For the strongest vision-capable models, reaching the first Pokémon Blue badge is no longer necessarily the hardest part.
Doing it efficiently is.
Against a human glitchless-style run of roughly 40,435 frames, or around 11 minutes:
In the online Pokémon setting, only a handful of models managed to reach the badge at all.
Kimi K3 stalled around the Pokédex in that comparison, although Patronus still places Kimi-K3 and Opus 5 among the strongest overall performers on harder games.
The message is clear:
Completion is becoming table stakes. Speed is not.
One of the benchmark’s more interesting findings comes from the offline-seed setting.
You might expect giving an agent an existing successful run to always help.
It does not.
On the first SuperTux stage, every agent capable of beating the level from scratch produced a worse time when starting from a mediocre reference run — in some cases by as much as 5.1×.
The models inherited the logic of the existing route and struggled to escape it.
That creates an interesting divide.
Weaker models often need the seed because they cannot discover a working route themselves.
Stronger models can succeed from scratch — but may become anchored to a bad solution when one is provided.
In other words, help is not free.
A seed does not only contain useful information.
It also contains someone else’s assumptions and mistakes.
Another striking result is how much the evaluation budget matters.
On shorter traces, giving an agent more opportunities to replay and score its proposed solution can sometimes matter more than which frontier model is doing the optimization.
Most agents do not suddenly discover revolutionary strategies.
They nibble.
They shave a few frames here, tighten a movement there, improve something they already understand.
What they rarely do is invent the completely new technique that deletes an entire minute from the run.
That distinction — between local optimization and genuine strategic discovery — may be one of SpeedrunBench’s most important contributions.
The most interesting thing about SpeedrunBench is not any individual playthrough.
It is what happens between them.
A human tool-assisted speedrun creator might watch a replay, notice that a jump lands two frames late, modify the input file, replay the game, and check the timer.
The offline agent is being asked to do essentially the same thing.
Its previous attempt becomes a research artifact.
It watches what happened, forms a theory about what could improve, changes the trace, and tests the result.
That is why Patronus describes the setup as an autoresearch loop.
The model is not merely a player anymore.
It is a speedrunner reviewing its own VOD.
And this is also where current agents begin to look both surprisingly capable and distinctly limited.
They are fairly good at local search.
They can refine a route they already understand.
They are much worse at deciding:
This entire strategy is wrong. Start over.
That behavior should sound familiar to anyone following coding agents.
A model may keep patching a failing test when what the system actually needs is a redesigned module.
It can optimize inside an existing frame.
Changing the frame itself is harder.
SpeedrunBench also fits neatly into Patronus AI’s broader strategy.
Founded in 2023 by Anand Kannappan and Rebecca Qian, Patronus has built much of its reputation around AI evaluation through projects including FinanceBench, Lynx, Glider, TRAIL, and MEMTRACK.
In June 2026, the company raised a $50 million Series B, bringing its reported total funding to roughly $70 million.
A major part of that vision centers on what Patronus calls digital world models: simulated environments where AI agents can repeatedly practice complex workflows before being trusted with real production systems.
SpeedrunBench is almost a perfect demonstration of that idea.
The environment is contained.
The objective is measurable.
The agent can experiment repeatedly.
And success does not end the task.
Once the agent finishes, the benchmark simply asks:
Can you do it better?
The paper’s authors span Patronus AI and the University of Washington: Yoshinari Fujinuma, Keisuke Kamahori, Abdelrahman Madkour, Varun Prashant Gangal, Monty Bichouna, Martyna Markiewicz, Shivani Jain, Duncan Curtis, Rebecca Qian, and Anand Kannappan.
Three ideas stand out from the launch.
First, completion benchmarks are becoming less useful once frontier models start consistently completing the task.
If one agent reaches Brock in 44 minutes and another effectively does it in 11, recording both as “successful” hides most of the interesting information.
Second, optimization and invention appear to be different capabilities.
The offline-seed and offline-scratch results make that particularly visible. Frontier models can often polish an existing solution. They struggle much more when the starting strategy is poor — or when no strategy exists at all.
That may turn out to be much closer to the challenge of real autonomous work than simply demonstrating that an agent can complete a predefined task.
Third, the environment around the model may matter almost as much as the model itself.
Giving GLM 5.2 hundreds of opportunities to test and refine its Super Mario Land run moved it to within a second of a human record.
Better search loops, replay systems, evaluation tools, and rewards may therefore unlock substantial gains without changing the underlying weights.
But Pokémon adds an important caveat.
More iterations cannot fully rescue a fundamentally inefficient strategy.
Search helps when you are near the right answer.
It helps much less when you are racing down the wrong road.
That may ultimately be the most interesting lesson in SpeedrunBench.
Human runners have spent decades proving that finishing a game is only the beginning. Once you know the route works, the real obsession starts: finding the unnecessary movement, the wasted input, the overlooked shortcut, the frame that did not need to be there.
AI agents now have a benchmark asking them to do the same.
The credits screen is no longer the finish line. The timer is.
| Game | Platform | Genre |
|---|
| Super Mario Bros. | NES | Platformer |
| Super Mario World / Super Mario Land | Game Boy | Platformer |
| Mario Kart 64 | Nintendo 64 | Racing |
| Pokémon Blue / Red | Game Boy | RPG |
| Civilization I | MS-DOS / SNES | Strategy |
| Wolfenstein 3D | MS-DOS / Browser | Shooter |
| Astray | Browser | Puzzle |
| SuperTux | C++ / Browser | Platformer |
| TuxeMon | Python / Browser | RPG |
| TuxCart | C++ / Browser | Racing |