Daily AI Paper

AI agents are starting to move from being passive helpers in science to something closer to research collaborators. In physics, that matters a lot, because real research is not just answering a question that already has a known solution. It means reading the literature, deciding what is worth doing, choosing methods, handling code or experiments, checking whether the result is actually trustworthy, and then writing it up in a way that would make sense to other physicists. The paper behind FrontierPhysics is about whether today’s frontier agents can really do that end to end. The motivation here is that most existing physics benchmarks do not test the full research loop. Some ask fixed questions with fixed answers. Others test isolated subtasks like reasoning, coding, or reproducing a paper when the solution path is already known. Those benchmarks are useful, but they often saturate quickly, and they do not tell us much about whether an agent can handle an open-ended project. Even paper reproduction benchmarks can be misleading, because the original paper gives away the structure of the solution. More recent benchmarks try to hide the answer by turning published work into a problem, but the authors of this paper argue that there is still a hard problem underneath: how do you design tasks that preserve genuine open-endedness without introducing accidental traps, and how do you judge whether a failure is a real scientific weakness or just a mismatch between the benchmark and a plausible research path? FrontierPhysics is meant to address that gap. It is an end-to-end benchmark built by physicists from their own research projects. That point matters a lot. The tasks are not synthetic toy problems; they come from real projects that the contributors actually worked on. The benchmark spans 56 tasks across nine physics domains, covering theoretical, computational, and experimental work. The contributors include Ph.D. students, postdocs, and research scientists, and the paper says the average contributor has around 420 citations. The authors also note that some of the underlying research has already appeared in journals such as Nature, Nature Communications, Physical Review Letters, and Physical Review X. So this is trying to benchmark something close to authentic research practice, not just puzzle solving. To understand the setup, it helps to think of each task as a small research assignment from a principal investigator. The agent gets a high-level prompt, some supporting files if needed, and a goal that is deliberately not over-specified. The prompt may mention important questions or directions to explore, but it does not provide a methodology or a solution path. The agent then has to do the whole job: review the literature, form a research plan, choose the right approach, run the computation or experiment, validate the outputs, and produce both deterministic artifacts and a scientific manuscript. The manuscript is submitted as paper.pdf and is expected to look like a real paper, with context, methods, results, and conclusions, and with the evidence from the run reflected faithfully. That combination of outputs is one of the most interesting parts of FrontierPhysics. The benchmark does not only ask, “Did you get the right number?” It also asks, “Did you do research in a way that would be publishable?” To evaluate that, the authors combine two kinds of grading. First, there is a deterministic verifier that checks structured outputs against reference values. That covers the objective side: computed quantities, simulation outputs, processed data, and similar artifacts. Second, there is rubric-based grading by reviewer agents. These rubrics include global criteria, such as whether the agent hallucinated results, reward-hacked, failed to synthesize the literature, or used evidence properly, and task-specific criteria written by the original task author, who knows what scientific insight really matters in that project. The benchmark design process is unusually careful. A task first has to pass a general quality review, where the contributor must prove provenance and authenticity. They need supporting material like reference papers or patent links, plus a description of the original project background, a rough timeline, and the amount of time spent. The project must have taken at least two weeks of research effort. The team also checks that the task description and rubric are genuinely human-authored and not just AI-generated text. Then comes domain peer review. The reviewer is an expert in a closely related field, and the checklist asks three important things: does the prompt avoid over-specifying the problem, does the rubric encode real physics insight rather than low-signal generic criteria, and do the observed failure modes reveal ambiguities or false positives and negatives? Contributors are required to run at least three local trials with strong agents and then analyze where those agents failed. Only after these steps does the task get approved by an admin team of physics Ph.D. researchers. That review pipeline is important because open-ended benchmarks are fragile. If you make the prompt too detailed, you accidentally turn research into a recipe-following exercise. If you make it too vague, you create false negatives where a good agent fails for the wrong reason. And if the rubric is shallow, you end up rewarding surface-level writing instead of scientific judgment. FrontierPhysics tries to reduce those problems by using the original researchers’ intuition to define what actually counts as success. The evaluation protocol is also more nuanced than a simple pass or fail. A task rollout is graded by combining the deterministic verifier with an agent-as-judge that inspects the trajectory files, code, outputs, and paper.pdf inside a separate sandbox. The reason for using an agent rather than a single judge call is that many rubric criteria depend on the whole process, not just the final paper. For example, you may need to inspect whether the agent really searched the literature, whether it validated results, or whether it produced unsupported claims. The rubric separates blockers from non-blockers. Blockers are essential scientific requirements: if one fails, the task cannot pass, no matter how good the rest looks. Non-blockers are scored on a three-level scale and weighted by importance. So a submission can be numerically correct but still score poorly if the manuscript is weak, poorly motivated, or inadequately supported. On the experimental side, the authors evaluate 22 frontier models in four different agent harnesses. Each model attempts all 56 tasks three times at its highest available inference effort, and four GPT models are also run at an even higher setting. The experiments are run in cloud sandboxes, using different tooling depending on the model family and resource needs. Some tasks run in lightweight sandbox environments, while others use Docker on Azure VMs with access to GPU resources when needed. Every rollout has a 10-hour timeout. The paper says these results are for FrontierPhysics version 0.1, and some tasks were re-run or regraded if later changes could affect the outcome. So what do the results show? The headline is that the benchmark is hard. Across 22 models in 26 configurations, no configuration passes more than 46 of the 168 rollouts. That means even the strongest systems are failing a large fraction of the time on end-to-end physics research tasks. But the more interesting finding comes from manual review of the trajectories. The authors conclude that the main weaknesses of frontier agents are not simply missing physics knowledge or making obvious execution mistakes. Instead, the bigger issues are insufficiently rigorous reasoning and limited expert intuition. In other words, the models may know a lot of physics and may be able to run code, but they still struggle to think like experienced researchers who know how to choose assumptions, test alternatives, recognize when a result is suspicious, and build a coherent scientific argument. That is a subtle but important result. It suggests that progress on scientific agents is not just about teaching them more facts or giving them better tools. It is about improving the quality of their research judgment. A model can be technically capable and still fail because it does not know which path is scientifically defensible. It may latch onto a plausible but weak idea, miss an essential validation step, or produce a manuscript that sounds polished while not actually supporting the claims. FrontierPhysics is designed to surface exactly those failures. There are, of course, limitations to keep in mind. The benchmark is still version 0.1, so it is an early snapshot rather than a final word. The tasks are drawn from a specific set of contributors and research styles, so the coverage, while broad across nine domains, is not the same as all of physics. The evaluation also relies partly on reviewer agents, which is practical for scale but still introduces its own judgment layer, even though the authors say they checked those grades against expert audits and on the authors’ own papers. And because the benchmark is built from real projects, there is always a tension between openness and fairness: too much openness can create ambiguity, while too much structure can reduce the task to a reproduction exercise. Still, FrontierPhysics is an important step. It moves the field away from asking whether an AI can solve isolated physics questions and toward asking whether it can actually participate in the research process. That is a much higher bar. And if agents are ever going to become useful scientific collaborators, that is the bar they will need to clear.