Daily AI Paper

Scientific discovery often starts with a surprise. You notice something that doesn’t fit your expectations, and then the real work begins: asking the right question, designing a test, and figuring out what rule could explain the anomaly. This paper asks whether large language models can do that on their own. Not just answer a question they were given, and not just follow a predefined benchmark, but freely explore a world, notice that something is odd, and uncover the hidden physical law behind it through experiments. That matters because a lot of the excitement around LLMs in science is about support roles: generating hypotheses, suggesting experiments, summarizing evidence, or helping with analysis. But autonomous discovery is a harder standard. A scientist doesn’t always begin with a task statement. Sometimes the key skill is curiosity itself: deciding what is worth investigating when nobody has told you what to look for. LongChang is built to test exactly that. The basic idea is to create simulated worlds where ordinary physics has been altered in one or more ways, but the model exploring the world is not told what changed. The explorer gets a physical environment with objects it can manipulate, along with observations of motion and position. It can push, throw, grasp, release, wait, mark reference points, and even build tools if needed. The challenge is to turn raw observations into a research question, run controlled experiments, and infer the altered law from the evidence. This is different from many existing law-discovery benchmarks. In earlier settings like PhysGym, NewtonBench, or DiscoverPhysics, the discovery target is usually specified in advance, or the system is already told what anomaly to explain. LongChang removes that crutch. The model has to notice the anomaly first. That makes the benchmark closer to open-ended scientific inquiry, because the hardest part is often not testing a hypothesis you already have, but deciding what deserves a hypothesis in the first place. LongChang also includes a tool-building component. That matters because some physical questions are hard to answer with direct poking alone. If the explorer wants better measurements or a way to isolate a force, it can design apparatus inside the same simulated world. A separate LLM checks whether the design is physically valid before it is used. So the system is not just exploring; it is also engineering instruments to support inquiry, which is a very realistic part of experimental science. There are three main stages in the framework: environment construction, autonomous exploration, and discovery evaluation. In environment construction, an LLM designer specifies the scene geometry, objects, physical program, and any altered law. These scenes are compiled into MuJoCo rigid-body simulations with manipulable balls and boxes, and sometimes colored floor tiles. The base set includes one normal control scene and 14 scenes, each with one altered physical law. The altered laws can depend on object properties, spatial relations, contact events, or past actions. The key point is that the explorer does not see the law definition. Only the evaluator does. The paper gives an example where balls rise under an upward force when they are close, without any upward launch. That kind of rule could produce surprising motion that an explorer has to interpret. Is it distance-based? Does it apply to all balls or only some? Is it triggered by contact, proximity, color, or something else? The explorer has to work this out from experiments. LongChang goes further by letting the environment evolve. Unresolved questions and corrected mistakes from earlier explorations are fed into scene designers, who can generate new scenes, refine existing ones, or combine mechanisms into more complex setups. This creates a full discovery loop without human intervention: explore, evaluate, update the scene, and explore again. That is an interesting design choice because it turns the benchmark into something more like a scientific program than a one-off test. During autonomous exploration, the model receives structured observations: object positions, motion, instrument readings, distances in bricks, and time in counts. It starts with a prompt that explicitly says to explore freely, with no research task and no hints about altered physics. The explorer then chooses what to investigate, records its observations and rationale, acts, and receives updated observations. If it wants to create a device, it can propose a design with geometry and mechanical connections such as joints and springs. If the design fails physical validation, the reviewer rejects it and the explorer can revise and resubmit. Successful tools are installed into the scene, and the explorer can then use them to gather better evidence. At the end, the model writes a report summarizing what it observed and what it thinks the law is. Importantly, the report is constrained to use only interpretations already recorded during exploration, and it has to preserve uncertainty. So the final answer is not supposed to be a polished guess detached from the evidence. It must be traceable to the actual experimental record. Evaluation is also split into two parts, which is crucial. The judge checks law correctness and evidence support separately. Correctness means the report identifies the altered law, including the target objects, conditions, triggers, and any constants that can be expressed in the observed units. Evidence support means the experiments in the record actually justify the claims at the stated level of certainty. This separation is important because a model can be right for the wrong reasons, or can gather good evidence but still fail to fully identify the law. The experiments evaluate 12 LLMs on the same 15 scenes, from fixed initial states, with up to 100 turns per scene. The paper also tracks exploration length, early stopping, action failures, and construction success, so we can see not just whether a model discovers the law, but how it behaves while trying. The headline result is stark. GPT-6-Astra is the only model that achieves a nonzero discovery rate, and even then it reaches 16.7%. The other 11 models score 0%. That means most models either fail to identify the altered law correctly, fail to support their claims with evidence, or both. But the deeper story is in the split between correctness and evidence. GPT-6-Astra has very high evidence support, 97.6%, but only 19.0% law correctness. In other words, it often produces reports that are well grounded in the observations, but the observations do not fully pin down the hidden law. The paper gives an example where a report supports attraction between red balls, but does not establish how that attraction depends on distance or whether it applies to other red spheres. That is a good illustration of partial discovery: the model notices something real, but stops short of the full rule. The opposite pattern appears for Claude-Opus-5. It identifies the altered laws in 16.7% of runs, but none of its reports pass the evidence support check. So it may guess the right law, but without enough experimental justification. This is one of the most interesting findings in the paper, because it shows that discovery is not just about being right. Scientific discovery also requires a defensible chain of evidence. The exploration statistics help explain why many models fail. Gemini-3.7-Flash stops early in every run, averaging only 10.9 turns, which leaves little room for follow-up experiments. The GLM models spend around 90 turns and successfully build apparatus in every run, but still make no discoveries. They also fail in more than 60% of build attempts, and failed actions account for a noticeable share of their turns. So even when a model is persistent and technically active, it may still not connect the observations into a coherent investigation. What do these results actually show? First, free exploration is hard. LLMs do not reliably turn anomalies into research questions on their own. Second, having the ability to act and build tools is not enough by itself. Third, the benchmark separates several skills that are easy to conflate: noticing an anomaly, designing a controlled test, interpreting the result, and writing a report that is both correct and supported. Different models can excel at one and fail at another. The paper also suggests a more subtle limitation: even when a model observes something unusual, it may leave the anomaly underinvestigated or only partially identify the law. That means the bottleneck is not always the first observation; often it is the follow-through. The model may see enough to suspect a pattern, but not enough to isolate the conditions under which the pattern holds. As for limitations, LongChang is still a simulated benchmark. The physics is controlled, the object set is limited, and the worlds are built from rigid-body scenes with a finite action interface. That makes the setting excellent for analysis, but it is still far from the complexity of real scientific discovery, where measurements are noisy, systems are messy, and the space of possible explanations is much larger. The paper also uses LLM-based reviewers and judges, which is practical and scalable, but as with any model-based evaluation pipeline, it raises familiar questions about reliability and calibration. Still, the main contribution is clear: LongChang turns autonomous physical discovery into a full cycle, from surprise to question to experiment to report to scene evolution. And the empirical result is sobering. In this setting, most LLMs do not independently uncover hidden physical laws. One model does so occasionally, but even then only rarely and often incompletely. So the paper’s answer to its opening question is: yes, in principle, an LLM can sometimes uncover a hidden law through free exploration, but current models are far from robust scientific discoverers. They can observe, they can act, and they can sometimes reason their way toward a law — but the leap from anomaly to genuine discovery remains very hard.