ProposalICLR 2027 workshop

How Far Are AI Agents from Becoming Computer Scientists?

Agents make progress when a person specifies the task and supplies a verifier on every answer. Real research often gives neither. This workshop is about what an agent does then.

We have proposed this workshop and its challenge for ICLR 2027. ICLR decides on November 29, 2026.

1

What to improve

  • Task given
  • Feedback verifiable

Today’s successes. How do harnesses, teams of agents, people in the loop and post-training get more out of the same setup?

2

What to trust

  • Task given
  • Feedback sparse

Most real research. How does an agent verify, replicate and try to refute a result it produced itself?

3

What to explore

  • No task
  • No feedback

The last mile. How does an agent choose a question, judge its novelty and decide when to drop it?

Invited speakers Confirmed

  • Alex DimakisUC Berkeley · Bespoke Labs
  • Azalia MirhoseiniStanford · Ricursive Intelligence
  • Ofir PressSWE-bench · ProgramBench
  • Yuandong TianRecursive Superintelligence
  • Matei ZahariaUC Berkeley · Databricks

Topics

  • Scaling at test time

    Harnesses, search, sampling and evolutionary methods

  • Multi-agent research

    Coordination, division of work and review among agents

  • Training research agents

    Post-training on research runs, RL and synthetic data

  • Verification and trust

    Replication, falsification and evidence agents make themselves

  • Failure modes

    Reward hacking, benchmark leakage, gains that do not transfer

  • Choosing questions

    Hypothesis generation, novelty and research taste

  • People and agents

    People as collaborators, verifiers and sources of taste

  • Benchmarks and environments

    Verifiable, open-ended computer science tasks

  • Applied research agents

    Systems research, kernel optimization, algorithm discovery

Challenge Problem to be announced

An open-ended problem in AI for AI R&D

Every team gets the same problem and the same budget. Let your research agents work on it alone, or collaborate with them. The winners present at the workshop.

Organizers