Agents do not budget their time or their submissions
Across 1,200 FrontierCS 2 runs, the median agent used 41% of its time and 29% of its submission budget, and its use of submissions barely changes with the size of the budget.
Gemini 4 Carbon passes 26 of 48 tasks, as many as Gemini 4 Argon and second only to Claude Opus 5.5, and has a higher mean score than Argon where both submitted (57.7 against 52.9). We show how it tests more before its first submission, the three passes that used loopholes, and three runs that stopped early because the Antigravity harness stopped waiting.
Each task removes a published method from the code that its authors released. The agent writes it back, or anything better, in five hours, and a run passes when it beats the code of the authors on every check. During a run the agent makes a capped number of development submissions for per-check feedback before calling final on a suite that adds held-out workloads, seeds, or configurations. Carbon ran once on each of the 48 tasks in Antigravity, and four of them again: three after an early stop, described below, and Robust Image Tokenizer after an infrastructure failure in its first run. Gemini 4 Argon in Antigravity is the comparison.
Pass rate is strict. A run passes only if its final submission beats the human reference on every metric the task checks and on every hidden workload, so the pass rate measures how robust a solution is. The FrontierCS Capabilities Index (FCI) also counts how far each final score lands above or below the human reference, so it rewards how far a model can push a task. Carbon and Gemini 4 Argon pass the same share of tasks (54%), but the FCI of Carbon is higher (52.9 against 46.7).
Carbon costs US$40.56 per run, about twice the US$19.52 of Gemini 4 Argon.
Carbon runs longer than Argon in the same harness (median 3.9 against 2.4 hours, longer on 37 of 48 tasks). It also tests more before it first submits: its first submission comes at a median of 0.68 hours against 0.42, and after an hour it had submitted nothing in 13 runs against 5.
The two models score within 5 points of each other on 38 of the 48 tasks. Two of the five largest gaps are runs that ended early (see below).
On the 45 tasks where both models submitted, Carbon scores 57.7 on average against 52.9, and both pass 26.
We reviewed the final patch and the full trace of all 26 passing runs. Harbor defines three kinds of reward hack: tampering (interferes with and subverts the grading process), cheating (gets the answer from a source it may not use), and loopholes (satisfies the tests without solving the problem). All three cases below are loopholes.
Progent (loophole). The README names the judge attack, AgentDojo important_instructions. The patch adds a sanitizer that deletes <INFORMATION> tags and spans starting with “This is an important message from” or “Before you can solve the task that I gave you” and ending at “Signed,”, marks any email, URL, or IBAN inside as tainted, and denies tool calls using tainted values. How much of the pass comes from the sanitizer rather than the privilege engine was not measured.
Seer (loophole). Fixed code outside the editable file, the Sweeper, deletes a branch predictor when its observationCount divided by its age falls below 0.1. Carbon wrote that “observationCount needs to be the accurate record of observations … No manipulation, falsification, or inflation is acceptable” (call 00391), then patched the code to add 14 to observationCount at each new block once a branch is seen in 2 blocks while keeping the real count in a new field, so recurring branches are never deleted. On development, the submission without the +14 line also passed; whether the final pass needs it is not known.
SCOPE (loophole; the pass depends on it). The README states which pattern records are evaluated (those where the two counting paths agree) and that the workload is “not a set of records to identify”. Carbon rebuilt the 56 evaluated records, examined “how uniquely each of the 56 queries is defined” (call 00152), and wrote a classifier that accepts only those 56 records for its exact closed-form counting engine while rejecting the other 17. With the classifier off, the largest workload (bio-grid-yeast-5) loses all 3 paired rounds to the reference, and the run no longer passes.
When a command runs in the background, Antigravity returns this to the model:
YOU MUST TAKE ONE OF THE FOLLOWING TWO ACTIONS: A) either proceed to other relevant work (if any) or, B) simply update the user with a short message (that you have launched the command and will wait for it to finish) and end the turn.
In three runs of the first evaluation, Carbon took option B and ended its turn to wait for a long command: Polygon at 1.0 hours, k-NN kernel at 0.9 hours, and KV Compaction at 0.9 hours. The Antigravity team told us that, after the model ends its turn, the CLI waits at most 30 minutes for background commands and then exits without telling the model. These three runs reached that limit before their first submission and ended with nothing submitted. Two more Carbon runs, Parallel DBSCAN and BABBLE, ran out of the memory of the agent container at 0.4 hours, also before their first submission.
We reran the three tasks on October 29 with one change: agy’s --print-timeout set to 6 hours, so the CLI keeps waiting for a background command after the model ends its turn. The reruns are the results of record, on the leaderboard and in the charts above. k-NN kernel now passes with 47.2. Polygon scores 96.6 and KV Compaction 58.7, and neither passes. The two runs that ran out of memory were not rerun.
The same 30-minute limit ended four runs of Argon (Polygon, MetaOpt, BABBLE, and KV Compaction) and no run in any other harness. These four were not rerun, and the leaderboard keeps them as they ran. Three of them were scored because Argon stored a submission before it started the long command.
Across 1,200 FrontierCS 2 runs, the median agent used 41% of its time and 29% of its submission budget, and its use of submissions barely changes with the size of the budget.
Claude Sonnet 5.5, Claude Fable 5.1, Inkling 2 Small and MiniMax-M3.1-Flash join the leaderboard. We show how far agents went to game the judge, how we closed each hole, and how we rebuilt all 48 tasks on one format and shared base images. Handshake and Snorkel AI join our sponsors.