Agents do not budget their time or their submissions
Across 1,200 FrontierCS 2 runs, the median agent used 41% of its time and 29% of its submission budget, and its use of submissions barely changes with the size of the budget.
ByWenhao Chai on behalf of FrontierCS 2 Team
FrontierCS 2 runs 25 agents, each a model inside a coding harness, on 48 research tasks, one run per agent and task. Every run gets 5 hours and a budget of 3 to 100 submissions for feedback on a dev set. Each submission returns a score and a set of checks. A run passes only if its final patch beats the human reference on every hidden workload and metric. The median agent used 41% of its time and 29% of its submission budget.
Share of the budget each agent used
Time (5 hours per task)Submissions for dev feedback
FrontierCSWenhao Chai
MiniMax-M3.1-Flash used the largest share of its time, Kimi K2.7 the largest share of its submissions, and Qwen 3.8 Max the smallest share of both. Across the 25 agents the two shares are unrelated, with a rank correlation of -0.02. The number of submissions also barely changes with the budget. On the 25 tasks with a budget of 100 submissions, the median run used 6. On the 7 tasks with a budget of 3, it used all 3.
We would not call this laziness. The agents have no budget awareness at all, of time or of other costs such as submissions.
The median run started its final evaluation 1.84 hours into its 5 hours. Many runs stopped while the dev checks still showed that the work was not done. In 780 runs the final patch had not passed every dev check, because a check failed, the patch was never sent to the dev set, or there was no final patch. Of these runs, 81% stopped with at least an hour left, at a median of 2.01 hours.
The dev checks predict the hidden result. Of the 420 runs whose final patch passed every dev check, 64.5% passed the hidden test. Of the other 780 runs, 5.4% did. Unused time and submissions are unused chances to check the work.
For people who build agents, this suggests showing the agent the time and submissions it has left, and having it spend them while a dev check fails. For benchmarks, it suggests reporting how much of the budget each agent uses next to its score.
All numbers come from the public FrontierCS 2 run dataset, metadata of 14 November 2026. Time used is the hour at which the final evaluation started, capped at 5 hours, averaged over the 48 runs of an agent and divided by 5 hours. Submissions used is the number of submissions sent for dev feedback divided by the budget of that task, averaged over the 48 tasks. The budget of each run is the one the platform reported to the agent. The analysis is observational, with one formal run per agent and task.
Claude Sonnet 5.5, Claude Fable 5.1, Inkling 2 Small and MiniMax-M3.1-Flash join the leaderboard. We show how far agents went to game the judge, how we closed each hole, and how we rebuilt all 48 tasks on one format and shared base images. Handshake and Snorkel AI join our sponsors.
Gemini 4 Carbon passes 26 of 48 tasks, as many as Gemini 4 Argon and second only to Claude Opus 5.5, and has a higher mean score than Argon where both submitted (57.7 against 52.9). We show how it tests more before its first submission, the three passes that used loopholes, and three runs that stopped early because the Antigravity harness stopped waiting.