Agents do not budget their time or their submissions
Across 1,200 FrontierCS 2 runs, the median agent used 41% of its time and 29% of its submission budget, and its use of submissions barely changes with the size of the budget.
Tag
2 posts tagged "Evaluation".
Across 1,200 FrontierCS 2 runs, the median agent used 41% of its time and 29% of its submission budget, and its use of submissions barely changes with the size of the budget.
Gemini 4 Carbon passes 26 of 48 tasks, as many as Gemini 4 Argon and second only to Claude Opus 5.5, and has a higher mean score than Argon where both submitted (57.7 against 52.9). We show how it tests more before its first submission, the three passes that used loopholes, and three runs that stopped early because the Antigravity harness stopped waiting.