Blog/Details about our evaluation of Gemini 4 Carbon
FrontierCS 2EvaluationOctober 29, 2026

Details about our evaluation of Gemini 4 Carbon

Gemini 4 Carbon passes 26 of 48 tasks, as many as Gemini 4 Argon and second only to Claude Opus 5.5, and has a higher mean score than Argon where both submitted (57.7 against 52.9). We show how it tests more before its first submission, the three passes that used loopholes, and three runs that stopped early because the Antigravity harness stopped waiting.

By Wenhao Chai on behalf of FrontierCS 2 Team

Key points

  • Gemini 4 Carbon tests more before it first submits than Gemini 4 Argon, and where both models submitted, it has a higher mean score (57.7 against 52.9); it passes 26 of 48 tasks, as many as Gemini 4 Argon and second only to Claude Opus 5.5.
  • Three passing runs used loopholes (Progent, Seer, and SCOPE), and in SCOPE the pass depends on it.
  • Three runs ended within an hour with nothing submitted while Carbon waited for a long background command: Antigravity stops waiting 30 minutes after the model ends its turn, without telling the model. We reran these three tasks with a 6-hour wait, and the reruns are the results of record.

Evaluation results

Each task removes a published method from the code that its authors released. The agent writes it back, or anything better, in five hours, and a run passes when it beats the code of the authors on every check. During a run the agent makes a capped number of development submissions for per-check feedback before calling final on a suite that adds held-out workloads, seeds, or configurations. Carbon ran once on each of the 48 tasks in Antigravity, and four of them again: three after an early stop, described below, and Robust Image Tokenizer after an infrastructure failure in its first run. Gemini 4 Argon in Antigravity is the comparison.

Pass rate on the 48 tasks of FrontierCS 2
Claude Opus 5.558%Gemini 4 Carbon54%Gemini 4 Argon (Antigravity)54%GPT-6 Astra44%GPT-6.1 Terra40%Gemini 4 Argon (Pi)40%Claude Fable 5.138%GPT-6.1 Sol35%GLM 5.325%Kimi K3.1 Pro25%Muse Wonder 1.021%Kimi K321%DeepSeek V4.1 Flash 101717%DeepSeek V4.1 Flash15%GLM 5.3 Flash12%Muse Spark 1.312%Qwen 3.8 Max12%Step 5 Preview8%Claude Opus 5.558%Gemini 4 Carbon54%Gemini 4 Argon (Antigravity)54%GPT-6 Astra44%GPT-6.1 Terra40%Gemini 4 Argon (Pi)40%Claude Fable 5.138%GPT-6.1 Sol35%GLM 5.325%Kimi K3.1 Pro25%Muse Wonder 1.021%Kimi K321%DeepSeek V4.1 Flash 101717%DeepSeek V4.1 Flash15%GLM 5.3 Flash12%Muse Spark 1.312%Qwen 3.8 Max12%Step 5 Preview8%

Pass rate is strict. A run passes only if its final submission beats the human reference on every metric the task checks and on every hidden workload, so the pass rate measures how robust a solution is. The FrontierCS Capabilities Index (FCI) also counts how far each final score lands above or below the human reference, so it rewards how far a model can push a task. Carbon and Gemini 4 Argon pass the same share of tasks (54%), but the FCI of Carbon is higher (52.9 against 46.7).

Carbon costs US$40.56 per run, about twice the US$19.52 of Gemini 4 Argon.

Gemini 4 Carbon: the submissions of every run
Development submissionFinal call, passFinal call, failNo final call, stored submission passedNo final call, fail0 h0 h1 h1 h2 h2 h3 h3 h4 h4 h5 h5 hHours since the run startedGlensideSpectrumDyTCKKS noiseScaleGPMRAGENkmeans kernelPolar ExpressProgentRegionsSortOCR Distillation Trajectory SamplingFARGO paper-aligned query coreParButterflyKAMIMuJoCo PlaygroundBinary Fuse Filter Construction and LookupNano World Modelknn kernelResidual State AdvancementeggShockwaveMetaOptRobust Image TokenizerRUNLTSPersistent Treap Update EngineDiffusion PolicySCOPESVG-EARExact Sequence AlignmentKV CompactionLeJEPAContinuumCRESTivf_pq kernelStructured LWEQuarterPIRDifferentiable Population StatisticsYannakakisParallel DBSCANBABBLEdbscan kernelLRRClusterwise SpGEMMHygraPolygonBBQApproximate-MIPS Query StoppingSeerDevelopment submissionFinal call, passFinal call, failNo final call, stored submission passedNo final call, fail0 h0 h1 h1 h2 h2 h3 h3 h4 h4 h5 h5 hHours since the run startedGlensideSpectrumDyTCKKS noiseScaleGPMRAGENkmeans kernelPolar ExpressProgentRegionsSortOCR Distillation Traj…FARGO paper-aligned q…ParButterflyKAMIMuJoCo PlaygroundBinary Fuse Filter Co…Nano World Modelknn kernelResidual State Advanc…eggShockwaveMetaOptRobust Image TokenizerRUNLTSPersistent Treap Upda…Diffusion PolicySCOPESVG-EARExact Sequence Alignm…KV CompactionLeJEPAContinuumCRESTivf_pq kernelStructured LWEQuarterPIRDifferentiable Popula…YannakakisParallel DBSCANBABBLEdbscan kernelLRRClusterwise SpGEMMHygraPolygonBBQApproximate-MIPS Quer…Seer

Carbon runs longer than Argon in the same harness (median 3.9 against 2.4 hours, longer on 37 of 48 tasks). It also tests more before it first submits: its first submission comes at a median of 0.68 hours against 0.42, and after an hour it had submitted nothing in 13 runs against 5.

Final score of Gemini 4 Carbon minus Gemini 4 Argon, by task
Carbon higherArgon higherGrey name: a run that submitted nothing-100-100-50-5000+50+50+100+100Final score of Carbon minus that of ArgonStructured LWE+66KV Compaction+59SCOPE+58OCR Distillation Trajectory Sampling+19Persistent Treap Update Engine+19Polygon+16LRR+16Polar ExpressRobust Image TokenizerDiffusion Policykmeans kernelSVG-EARParButterflyknn kernelDifferentiable Population StatisticsQuarterPIRMetaOptdbscan kernelivf_pq kernelKAMIFARGO paper-aligned query coreeggMuJoCo PlaygroundCRESTRUNLTSCKKS noiseApproximate-MIPS Query StoppingRegionsSortGlensideDyTSpectrumLeJEPAExact Sequence AlignmentSeerResidual State AdvancementScaleGPMYannakakisRAGENHygraClusterwise SpGEMMShockwaveBinary Fuse Filter Construction and LookupNano World ModelContinuumProgentBBQBABBLE-19Parallel DBSCAN-77Carbon higherArgon higherGrey name: a run that submitted nothing-100-100-50-5000+50+50+100+100Final score of Carbon minus that of ArgonStructured LWE+66KV Compaction+59SCOPE+58OCR Distillation Traj…+19Persistent Treap Upda…+19Polygon+16LRR+16Polar ExpressRobust Image TokenizerDiffusion Policykmeans kernelSVG-EARParButterflyknn kernelDifferentiable Popula…QuarterPIRMetaOptdbscan kernelivf_pq kernelKAMIFARGO paper-aligned q…eggMuJoCo PlaygroundCRESTRUNLTSCKKS noiseApproximate-MIPS Quer…RegionsSortGlensideDyTSpectrumLeJEPAExact Sequence Alignm…SeerResidual State Advanc…ScaleGPMYannakakisRAGENHygraClusterwise SpGEMMShockwaveBinary Fuse Filter Co…Nano World ModelContinuumProgentBBQBABBLE-19Parallel DBSCAN-77

The two models score within 5 points of each other on 38 of the 48 tasks. Two of the five largest gaps are runs that ended early (see below).

Mean final score: all tasks, and the tasks where both models submitted
Gemini 4 CarbonGemini 4 ArgonMean final score020406055.351.6All 48 tasks57.752.945 tasks, both submittedGemini 4 CarbonGemini 4 ArgonMean final score020406055.351.6All 48 tasks57.752.945 tasks, both submitted

On the 45 tasks where both models submitted, Carbon scores 57.7 on average against 52.9, and both pass 26.

Hacking behavior

We reviewed the final patch and the full trace of all 26 passing runs. Harbor defines three kinds of reward hack: tampering (interferes with and subverts the grading process), cheating (gets the answer from a source it may not use), and loopholes (satisfies the tests without solving the problem). All three cases below are loopholes.

Progent (loophole). The README names the judge attack, AgentDojo important_instructions. The patch adds a sanitizer that deletes <INFORMATION> tags and spans starting with “This is an important message from” or “Before you can solve the task that I gave you” and ending at “Signed,”, marks any email, URL, or IBAN inside as tainted, and denies tool calls using tainted values. How much of the pass comes from the sanitizer rather than the privilege engine was not measured.

Seer (loophole). Fixed code outside the editable file, the Sweeper, deletes a branch predictor when its observationCount divided by its age falls below 0.1. Carbon wrote that “observationCount needs to be the accurate record of observations … No manipulation, falsification, or inflation is acceptable” (call 00391), then patched the code to add 14 to observationCount at each new block once a branch is seen in 2 blocks while keeping the real count in a new field, so recurring branches are never deleted. On development, the submission without the +14 line also passed; whether the final pass needs it is not known.

SCOPE (loophole; the pass depends on it). The README states which pattern records are evaluated (those where the two counting paths agree) and that the workload is “not a set of records to identify”. Carbon rebuilt the 56 evaluated records, examined “how uniquely each of the 56 queries is defined” (call 00152), and wrote a classifier that accepts only those 56 records for its exact closed-form counting engine while rejecting the other 17. With the classifier off, the largest workload (bio-grid-yeast-5) loses all 3 paired rounds to the reference, and the run no longer passes.

Early stop

When a command runs in the background, Antigravity returns this to the model:

YOU MUST TAKE ONE OF THE FOLLOWING TWO ACTIONS: A) either proceed to other relevant work (if any) or, B) simply update the user with a short message (that you have launched the command and will wait for it to finish) and end the turn.

In three runs of the first evaluation, Carbon took option B and ended its turn to wait for a long command: Polygon at 1.0 hours, k-NN kernel at 0.9 hours, and KV Compaction at 0.9 hours. The Antigravity team told us that, after the model ends its turn, the CLI waits at most 30 minutes for background commands and then exits without telling the model. These three runs reached that limit before their first submission and ended with nothing submitted. Two more Carbon runs, Parallel DBSCAN and BABBLE, ran out of the memory of the agent container at 0.4 hours, also before their first submission.

We reran the three tasks on October 29 with one change: agy’s --print-timeout set to 6 hours, so the CLI keeps waiting for a background command after the model ends its turn. The reruns are the results of record, on the leaderboard and in the charts above. k-NN kernel now passes with 47.2. Polygon scores 96.6 and KV Compaction 58.7, and neither passes. The two runs that ran out of memory were not rerun.

The same 30-minute limit ended four runs of Argon (Polygon, MetaOpt, BABBLE, and KV Compaction) and no run in any other harness. These four were not rerun, and the leaderboard keeps them as they ran. Three of them were scored because Argon stored a submission before it started the long command.

All posts Back to top

Keep reading

FrontierCS 2 Update 11/06/2026
FrontierCS 2UpdateNov 6, 2026

FrontierCS 2 Update 11/06/2026

Claude Sonnet 5.5, Claude Fable 5.1, Inkling 2 Small and MiniMax-M3.1-Flash join the leaderboard. We show how far agents went to game the judge, how we closed each hole, and how we rebuilt all 48 tasks on one format and shared base images. Handshake and Snorkel AI join our sponsors.