Daily AI Paper

Computer-use agents are supposed to help us work with software the way a human would: by seeing what is on the screen, deciding what to do next, and carrying out the action. But in practice, these agents are limited not just by the model inside them, but by the interface they are given. If the only way to edit a spreadsheet is by clicking cell by cell, then even a strong model is forced into a clumsy workflow. If the only way to inspect a file is to print thousands of lines through a shell, then the agent wastes time and context just finding one function. This paper argues that interface design itself should become something an agent can learn and improve, rather than a fixed human decision made once before deployment. That matters because computer use is open-ended. New applications appear, workflows change, and model capabilities keep improving. Every time the task family shifts, humans have to inspect failures, invent new helpers, rewrite observation formats, or add memory mechanisms. That is expensive and does not scale. The authors point out a second problem too: when the interface is mismatched to the task, agents often compensate with more test-time compute instead of a better design. That can raise benchmark scores, but it is often a brittle workaround. And a third limitation is that even when agents learn reusable skills from experience, those skills are still restricted by the original interface. They can rearrange what the agent already can do, but they cannot expand what the agent can observe or request. To address this, the paper introduces Agent S4, described as an autonomous computer-use scientist. Its job is not to solve tasks directly, but to study how task agents behave and to evolve better interfaces from those experiences. The key idea is to maintain a persistent repository containing both executable interfaces and the knowledge for when and how to use them. So instead of a single fixed GUI-only or shell-only setup, the repository can contain a mixture of desktop-wide, application-specific, cross-application, and task-specific tools, along with notes, workflows, and capability descriptions. A useful way to think about the setup is that the task agent itself stays fixed. The model and the harness do not change. The harness still handles model calls, permissions, task budgets, and execution. What changes is the repository the agent uses. The agent interacts with a computer through that repository, and the resulting trajectories are what the system learns from. The authors call these reusable, experience-built components Evolving Agent Interfaces. The background here is familiar if you have worked with computer-use systems. A standard GUI interface gives you clicks and keystrokes and returns screenshots. A shell interface gives you commands and text output. But many tasks would be easier with richer, more structured tools. For example, a spreadsheet helper might expose ranges of cells and allow bulk edits; a browser helper might return structured data instead of just a screen image; a programmatic interface might let an agent query or modify a file directly. The paper’s claim is not that one interface is universally best, but that the right interface depends on the task, and that experience can reveal what should be added, revised, or removed. The method is population-based evolution over repositories. Each repository is a candidate design. The population starts from a seed repository, and then repeatedly produces descendants by mutation or crossover. Mutation means taking one parent repository and adding, revising, or removing interfaces and supporting knowledge. Crossover means combining two parent repositories into one coherent descendant, choosing between overlapping interfaces and porting complementary capabilities. The meta-agent doing this work is itself a model running in a controlled environment. It can read training trajectories from task agents, inspect what happened, and form hypotheses about why the agent struggled or wasted effort. If it sees repeated cell-by-cell spreadsheet edits, it may infer that a bulk-edit operation would help. If it sees that an existing helper is being used poorly, it may revise the usage guidance rather than the code. It then implements these changes through file, shell, and code interfaces. A very important design choice is how the paper prevents overfitting to the exact tasks that generated the training trajectories. The meta-agent sees training experience, but validation tasks are hidden from it. Validation instructions, trajectories, diagnostics, and scores are never shown during construction. Validation performance is used only for selection: it decides which repositories survive and get developed further. That separation is meant to favor changes that generalize beyond the seen tasks, rather than just memorizing quirks of the training set. The paper also describes the repository as more than just code. It includes executable interfaces plus supporting knowledge: capability descriptions, reusable workflows, configuration, and notes about when a tool should be used. That matters because a good interface is not only about what operations exist, but also about helping the agent choose the right one at the right time. For experiments, the authors evaluate Agent S4 on two benchmarks: OSWorld V2 and ScienceBoard. OSWorld V2 is a broad computer-use benchmark with 108 tasks. ScienceBoard covers more niche scientific applications. The task agent in the main OSWorld experiment uses Qwen3.8-27B with the evolved repository. The paper also compares against fixed-interface baselines, including GPT-5.5 and Claude Opus 4.8, and it reports cost comparisons as well. The main result on OSWorld V2 is that Qwen3.8-27B with the evolved repository reaches 21.3% binary success. That is presented as a new state of the art among open-source computer-use agents. It is also 3.8 times the same model with fixed interfaces. Compared with the proprietary baselines, it is above GPT-5.5 at 13.0% and slightly above Claude Opus 4.8 at 20.6%, while costing about 4.3% and 1.5% of their respective costs. So the paper is not just claiming a small gain; it is showing that interface evolution can materially change the performance of the same underlying model. On ScienceBoard, the evolved repository improves success rate from 41.4% to 58.6% with the same model. That is important because it suggests the benefit is not limited to one benchmark style. The authors interpret this as evidence that useful interfaces can be evolved from purely synthetic, verifier-free task experience. In other words, the system does not need ground-truth success labels or an external task verifier to make progress. It can learn from trajectories alone, using the meta-agent’s analysis and the validation split for selection. The paper also compares against evolutionary baselines such as linear evolution, and it studies the effect of giving the meta-agent a scratch virtual machine for testing its evolved interfaces. While the detailed numbers are not the central story here, the point is that the authors are testing whether the evolution process itself matters, and whether a more capable meta-agent environment helps it discover better designs. Qualitatively, they report that the evolved repositories change both what the agent observes and how it acts. These changes span multiple levels: desktop-wide interfaces, application-specific helpers, cross-application routines, and task-specific tools. That is an important takeaway. The improvement is not just “add one better command.” It is a broader redesign of the action and observation space. What do these results actually show? First, they show that fixed interfaces are not a harmless implementation detail. The same model can perform very differently depending on whether it has a generic GUI or a better tailored interface repository. Second, they show that experience can be turned into reusable interface design, not just into a policy that repeats past behavior. Third, they suggest that interface evolution is complementary to model scaling and harness engineering. Better models still matter, but there may be another path to gains: let the agent redesign the way it interacts with the computer. There are also clear limitations to keep in mind. This is still an evolutionary system, so it depends on synthetic experience, population management, and a meta-agent that can write code and reason about trajectories. That means it introduces its own engineering complexity. The paper also does not claim that the meta-agent is magically discovering interfaces from nothing; it is operating within a fixed harness and a specific repository representation. So the approach is powerful, but it is not unconstrained. Another limitation is that the benefits are benchmark-dependent. The paper shows strong results on OSWorld V2 and ScienceBoard, but those are still benchmark environments, not the full diversity of real-world computer use. And because the final selection relies on validation performance, the quality of the validation split matters a lot. If the validation tasks are poorly chosen, evolution could still drift toward narrow specialization. Still, the broader message is compelling: instead of treating interfaces as static human artifacts, we can treat them as part of the agent’s learning problem. The paper’s title captures that idea well. The interfaces themselves evolve, and that evolution becomes a source of capability improvement alongside better models and better harnesses. The authors have open-sourced the code, which should make this line of work easier to study and extend.