FrontierCS 2: A Benchmark for Computer Science R&D
48 open-ended tasks, each built by deleting the key contribution of a computer science paper from its released code.
Research
Papers from the FrontierCS team.
FrontierCS: Evolving Challenges for Evolving Intelligence
The benchmark paper: how the problems are built and scored, and how models compare with human experts.
FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale
Turns closed-ended coding problems into open-ended ones for training, with results on FrontierCS and ALE-bench.
LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?
Competitive-programming problems from live contests, annotated by Olympiad medalists, to find where LLMs still fall short of experts.
When Agents Slow Down: Understanding LLM Agents’ Test-Time Strategies via Elo-per-token Analysis
Follows the best solution as the token budget grows, and finds where an agent falls behind independent sampling.
EasyPPO: Stabilizing the Critic Is Key
A stable PPO for language models, trained with FrontierCS’s continuous score among its rewards.
Built on FrontierCS
Research that uses the FrontierCS benchmarks.
Open source
- Reef: Infrastructure for Continually Self-Improving AgentsHuman-Agent-Society · 7.7k stars
Connects agent inference, feedback, learning and versioned delivery. Its Guidance-TTT recipe runs FrontierCS Polyomino Packing with the FrontierCS judge.
Post-training
- Evolution Fine-Tuning: Learning to Discover Across 371 Optimization TasksLee et al.
Turns evolutionary search runs into training data; nearly half of its 371 tasks derive from FrontierCS. On held-out problems its 9B model averages 46.01, against 32.46 for the base model.
- AREX-2: Advancing Self-Improving Agents through Long-Horizon Reflective TasksQian et al.
Trains a 27B agent on long trajectories in which it keeps improving its own solution. On 188 FrontierCS tasks it scores 70.7, 16.0 points above the strongest open-weight baseline.
Test-time scaling
- Scaling Discovery through Test-Time CommunicationPark et al.
Compares a team of agents that share findings with the best of as many independent agents. On Polyomino Packing the team reaches 0.945, above the best score reported before it.
- Evolving in Thought Space: Training a Small Model at Test Time Unlocks Better DiscoveriesJiang et al.
Trains a small guidance model with reinforcement learning during the search; a frozen model writes the code. On Polyomino Packing it reaches 91.89, against 89.40 for CORAL and 89.10 for the human reference.
- Fork-and-Flush: Escaping Idea Basins in Autoresearch AgentsCai et al.
Periodically forks an agent’s run and clears its context so it leaves a plateau. Nine of its thirteen tasks come from FrontierCS; its normalized average is 0.78, against 0.47 for a single run.
- Stateless Language Agents: Scaling Long-Horizon Automated ResearchZhang et al.
Keeps the research state in the harness and builds a fresh context for every agent call. On a FrontierCS algorithm-design problem it scores 59.5 at 200M tokens, against 58.0 for the strongest baseline.
- Scaling Discovery Requires ForgettingTan et al.
Periodically clears all agent sessions and the candidate history, and restarts a communicating team from its best candidate. On FrontierCS algorithmic problem 171 the team’s mean best score is 80.8, against 70.2 for four parallel agents at the same token cost.
Agent harness
- AdaEvolve: Adaptive LLM Driven Zeroth-Order OptimizationCemri et al.
Compares evolutionary search methods on the FrontierCS algorithmic problems with 50 GPT-5 calls each. AdaEvolve averages 61.33, against 50.75 for OpenEvolve and 20.64 for a single call.
- EvoX: Meta-Evolution for Automated DiscoveryLiu et al.
Evolves the search strategy during a run as well as the solutions. On the FrontierCS algorithmic problems, with 100 iterations each, it reaches a mean of 62.6 and a median of 75.5.
- CORAL: Towards Autonomous Multi-Agent Evolution for Open-Ended DiscoveryQu et al.
Replaces fixed evolutionary rules with long-running agents that share memory. On Polyomino Packing, four agents reach 89.4% coverage, against 56.0% for a single attempt.
- Don’t Gamble, GAMBLe: An Analytical Framework for AI-Driven Research SystemsEllis and Castro
Splits a discovery system into generator, assessor, mechanism and budget, and tests each part in 760 runs on three FrontierCS problems. On Polyomino Packing, GPT-OSS-20B reaches a median of 45.0, against 21.5 for Claude Opus 4.6.
- Repo-To-Skill: Distilling GitHub Repositories Into AI4AI SkillsChen et al.
Distills GitHub repositories and papers into verified skills for a research agent. On 188 FrontierCS tasks the skills raise Codex from 70.63 to 77.14.
- OptiCom: A Unified Framework for State-Conditioned Composition in LLM-Driven OptimizationWei et al.
Composes search operators according to the state of the run. Averaged over 172 FrontierCS problems, its best of five runs scores 0.693, against 0.632 for the strongest baseline.
- LabBook: Harnessing Experimental History for Efficient LLM-Driven DiscoveryYuan et al.
Reuses the history of earlier experiments to guide the search. On 49 FrontierCS algorithmic problems its mean best score is 6.13 points above the strongest baseline with DeepSeek V4 Flash.
Evaluation infra
- Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic EvaluationShi et al.
Ports benchmarks to one agent-evaluation format and checks each port against the original. Its catalog has a verified adapter for the FrontierCS algorithmic problems, checked on 10 of the 172.