Daily AI Paper

Large language model agents are increasingly being asked to do more than answer questions. They’re being used as tool-using assistants that can search databases, update records, place orders, manage calendars, and carry out multi-step workflows in stateful environments. That makes the problem much harder than ordinary text generation, because the agent has to choose the right tool at the right time, remember what has already changed, and adapt when the environment is different from the one it saw during training. This paper is about how to train those agents effectively when the training data itself is synthetic. The central question is not just how much data to generate, but what kind of data composition actually leads to better generalization. If you have a fixed budget for synthetic tasks, should you spend it on more tasks using the same tools? On more diverse tools? On the same tasks but with different database states? Or on making the environments more complex? The paper argues that these choices matter a lot, and that their importance changes as you scale up training. That matters because real tool-use data is scarce. Multi-step interactions with stateful tools are not common in ordinary internet text, and collecting them manually is expensive. So a lot of recent work has turned to LLMs to synthesize training data. Some approaches generate demonstrations over fixed APIs. Others simulate complete tool trajectories with an LLM, including the tool responses. That can work, but as the environment gets more complex, simulating every response and state transition reliably becomes difficult. A different line of work generates executable environments, where tools are backed by code and persistent state. This paper builds on that idea, but it focuses on a deeper issue: once you can synthesize environments, how should you allocate your budget across different kinds of environment diversity? To study that, the authors introduce SynthAPI, a controllable pipeline for generating multi-step tool-use environments. It has two main stages. First, it generates apps: coherent bundles of tools that operate over structured databases. These are implemented in Python from scratch by a “builder” language model. The authors define a broad taxonomy of app types, including shopping, music, productivity, finance, fitness, messaging, utilities, and social. For each app, the model creates metadata describing the app, a database schema that defines the needed data types, and an API layer with functions that read and modify the database. The key idea is that the tools are not isolated; they form realistic workflows with dependencies, like a shopping app where searching products, inspecting reviewers, applying promo codes, and creating orders all interact with shared state. The second stage is task generation. A task here is not just an instruction. It is a tuple consisting of the instruction itself, the starting database state, a verifier, and an oracle solver. The instruction is generated by sampling an app and selecting a subset of its functionalities. The verifier is executable code that checks whether the task has been completed correctly from the current database state. The starting database state is built in layers. First, the system creates the minimum core data needed for the task to be feasible. Then it adds distractor data in additional batches to increase realism and complexity. Finally, the system can generate an oracle solver that solves the task from that starting state. What makes SynthAPI especially important for this paper is that it exposes explicit controls over three axes of diversity: tools, tasks, and data profiles. In plain terms, that means the authors can vary how many different apps and APIs the agent sees, how many different tasks it gets for those tools, and how many different initial database states those tasks are paired with. That lets them ask a question that earlier synthetic-environment work did not cleanly answer: if you have a fixed amount of training compute or a fixed data budget, what composition of synthetic environments gives the best downstream generalization? The experimental study is large. The authors vary SynthAPI’s generation controls across 458,752 tasks and train 849 models of four different sizes. They then evaluate how well a model trained on a given synthetic environment mix generalizes to downstream tool-use benchmarks. The point is not just to compare one recipe against another, but to derive a scaling law: a predictive relationship that maps environment composition and scale to downstream performance. The results show a clear pattern. As the data budget grows, it becomes more valuable to expand tool diversity than to keep generating more tasks using the same tools. In the same way, generating the same tasks over more varied database states becomes more useful than simply adding new tasks within a narrow environment. The paper also finds that increasing data complexity helps more at larger budgets. So the best use of extra data is not just “more of the same.” It is more diverse tools, more diverse database profiles, and richer stateful settings. The authors turn this into an environment-composition-dependent scaling law. They hold out the runs in the top 25 percent of training FLOPs and test whether the law can extrapolate to them, including runs with up to 16 times the largest FLOP budget used for fitting. It does. The reported mean absolute error on normalized downstream score is 0.0285, compared with 0.1706 for a scaling model that uses only the number of tasks and ignores environment composition. That is a strong sign that composition matters, not just raw task count. They also use the scaling law to choose a better training recipe. With that recipe, SynthAPI transfers well across four evaluation benchmarks, including AppWorld, BFCL, and ASTRA-Bench. At a fixed data budget, the composition chosen by their analysis outperforms a simpler strategy that just scales task count, and it does so across all model sizes and benchmarks they test. The paper also reports that agents trained on SynthAPI data outperform agents trained on prior large-scale tool-use agent data by 30.5 percentage points on average across the three benchmarks. Against strong per-benchmark synthetic-data baselines, SynthAPI is better by 7.6 to 11.6 points on AppWorld Test-Challenge and by 2.0 to 3.7 points on BFCL. What should we take from that? First, if you want tool-use agents that generalize, you should not think of synthetic data as a single blob of “more examples.” The composition of the environment matters. Second, the value of diversity changes with scale. Once you have enough budget, it is better to spend it on broader tools and broader database states than on repeating similar tasks. Third, executable, database-backed environments appear to be a more stable way to scale stateful tool-use training than relying on fully simulated tool responses. There are also important limitations to keep in mind. SynthAPI still depends on LLM-generated apps, schemas, instructions, verifiers, and oracle solvers, so the quality of the generated environment remains tied to the generator. The paper also focuses on structured, database-backed workflows, which are highly relevant to current benchmarks but do not cover every kind of tool use in the wild. And while the scaling law is predictive within the studied regime, any such law is only as good as the space it was fit on; it is a powerful guide, not a guarantee for every future environment or benchmark. Still, the main contribution is clear. This paper does not just show that synthetic tool-use data can scale. It shows that how you scale it matters, and it provides a concrete way to predict and optimize that choice. For anyone building agentic systems, that is a useful shift: from asking how many synthetic tasks to generate, to asking what mixture of tools, tasks, and environment states will actually teach the agent to generalize. The paper’s broader implication is that environment generation itself becomes a first-class part of agent training. Once you move from static question-answering to interactive workflows, the training set is no longer just a list of prompts and labels. It is an ecosystem of actions, state transitions, and constraints. That changes what it means to scale well. A model can memorize repeated task patterns if the environment is too narrow, but it is much harder to overfit to a genuinely varied set of tools and database states. That is why the authors’ emphasis on diversity is so important: it pushes the training distribution closer to the kinds of shifts that real agents face after deployment, when a new app, a new schema, or a new initial state can break brittle policies. Their task construction also helps explain why executable environments are appealing. Because the tasks come with verifiers, the system can automatically check success rather than relying on approximate human judgments. And because the environments are backed by Python code and structured databases, the agent is forced to interact with consistent state rather than a hand-waved simulation of tool outputs. In practice, that makes it easier to generate large volumes of supervision while still preserving the logic of multi-step tool use. It also creates a setting where one can study not only whether an agent succeeds, but how the training distribution affects the kind of reasoning it learns: whether it learns to search, inspect, update, recover from irrelevant data, and chain tools in the right order. This kind of work is likely to influence how synthetic agent datasets are built going forward. Instead of asking a generator for a fixed number of random tasks, practitioners may start designing data-generation pipelines with explicit knobs for environment breadth, task variety, and state complexity. That could matter for many downstream systems, from customer-support agents that need to query account databases, to enterprise assistants that edit records across multiple internal tools, to personal assistants that manage calendars, messages, and shopping workflows. In those settings, the challenge is not only solving known actions, but generalizing to unfamiliar combinations of tools and states. A scaling law that predicts which composition of synthetic data helps most is valuable because it turns dataset design into something more principled and budget-aware. The comparison to prior work is also telling. A lot of earlier synthetic tool-use efforts focused on generating more trajectories, which is a natural first step. But this paper suggests that quantity alone is not the main story once you are already operating at scale. If two training runs use the same number of tasks, the one with more diverse tools and more varied initial states can be substantially more useful. That is a subtle but important lesson for anyone trying to improve agent generalization with limited compute. It means the best data is not always the largest pile of examples; it is the best allocation of variation. So the contribution here is both methodological and conceptual. Methodologically, SynthAPI provides a way to synthesize realistic, stateful, executable environments with explicit control over what changes. Conceptually, the paper shows that tool-use agents obey scaling behavior that depends on environment composition, not just on raw dataset size. That gives researchers a roadmap for building stronger agents: vary the right things, measure the effect systematically, and use the resulting scaling law to decide where the next unit of data should go.