Research notes, benchmark releases, and agent evaluation insights.
Claude Sonnet 5.5, Claude Fable 5.1, Inkling 2 Small and MiniMax-M3.1-Flash join the leaderboard. We show how far agents went to game the judge, how we closed each hole, and how we rebuilt all 48 tasks on one format and shared base images. Handshake and Snorkel AI join our sponsors.
Twelve models tried to beat the code that the SVG-EAR authors released for their ECCV 2026 sparse-attention method, and none passed. We show what each run tried, which of its 48 checks it missed, and by how much.
BBQ, an NSDI 2024 circuit, picks the next packet a network card sends. Gemini 4 Argon and Claude Opus 5.5 each gave it a faster clock than its authors did, one with the same memory and one with registers in its place. We build the circuit up from a 16-priority example to show both.
QuarterPIR, a EUROCRYPT 2024 scheme, lets a client read a database without revealing what it reads. Muse Wonder 1.0 cut its bytes per lookup by 27%, with privacy unchanged. We build the scheme up from a 16-entry example to show how.
Agents can spend test-time compute by trying, observing, and revising. We derive an Elo reference for repeated sampling, then show that in a 2022 two-week coding marathon, current agents plateau within 24 hours while top humans keep improving.
FrontierCS 2 extends open-ended evaluation from static sandboxed tasks to Harbor-based feedback loops and repo-level environments.
We release FrontierSmith, a system that converts closed-ended coding problems into open-ended optimization tasks for training long-horizon coding agents.
We integrate FrontierCS into Harbor and release a preview long-horizon agent leaderboard on 178 open-ended algorithmic tasks. Kimi K2.6 and Claude Code Opus 4.7 show similar headline capability, but very different failure modes.
We placed an open-ended optimization problem in CALICO, UC Berkeley's official programming contest. To score, you had to beat a frontier AI. Out of 2,000+ contestants, only one submission surpassed the strongest AI agent.
FrontierCS is an open benchmark that facilitates evaluation of evolving agents. We demonstrate how researchers build ALMA to automatically learn continual learning AI researchers on FrontierCS.
Evolving agent systems are advancing fast, but evaluation hasn't kept up. We show how FrontierCS enables comprehensive, large-scale benchmarking of evolving agents—moving beyond small case studies for comparison at scale.
Modern LLMs claim superhuman algorithmic abilities, but what happens when there is no strict verifier? We analyze how multi-turn 'optimization' in FrontierCS exposes the cognitive ceiling and catastrophic failures of AI in open-ended problem solving.
FrontierCS scores solutions on a continuous scale across heterogeneous hardware. This post explains the evaluation architecture behind the leaderboard: hash-based resume, resource-grouped clusters, pinned environments, and the challenges ahead for agentic submissions.
We are releasing FrontierCS 1.0, a major update to our open-ended Computer Science benchmark. This release expands FrontierCS to 240 tasks across both the algorithmic and research tracks. We also introduce a new Elo-based leaderboard, along with full execution traces of model solutions to enable deeper analysis and reproducibility.
