Daily AI Paper

Teleoperation is still the default way to collect robot demonstrations, and for good reason: it gives you data that is faithful by construction. The robot really executed the action, and the camera really saw the scene where the policy will later operate. But that fidelity comes with a big cost. Every episode needs a robot, a human operator, and often a reset after failure. So the cost grows linearly with the amount of data, and then it starts over again for every new task, every new scene, and even every new viewpoint. That scaling problem matters because modern robot policies, especially vision-language-action systems, are hungry for lots of demonstrations. The field has tried to reduce the burden in a few ways. Some methods multiply a small number of demonstrations in simulation. Others use language models to help write tasks or rewards. Real-to-sim-to-real pipelines reconstruct the physical scene and then use simulation for training. And video generation models can make robot footage look realistic. But each of those directions leaves something unresolved. If a person still has to engineer the task, you have not really removed the labor. If the scene is reconstructed imperfectly, then cloth, displays, lighting, and other tricky appearance details still look synthetic. And if a video model generates motion itself, then you may lose exact action labels, which are crucial for training a robot policy. This paper asks a sharper question: can a coding agent take over the teleoperator’s job, while a simulator provides the exact actions and a video model provides the appearance? The answer they propose is DataFarm, an agentic data farm for producing what they call real-ready robot data. The key idea is to split the problem in two. The simulator handles the parts that must be exact: actions, states, contacts, and task execution. A video model, conditioned on real images from the target scene, handles the parts that must look real: the camera views and scene appearance. The output is a synchronized multi-view dataset that can be trained on directly, just like teleoperated data, and then deployed on the real robot. To make that work, DataFarm starts from a very modest input: one calibration episode and a data request. The calibration episode is a short multi-view recording of the robot in the target scene. It is not used as a source of actions; instead, it tells the system what the scene and cameras look like. The request also includes a natural-language description of the task, the number of episodes desired, and whether the farm should reproduce the recorded scene or also allow appearance variation, such as a new tablecloth or colored lighting. From there, an LLM-based coding agent takes over. It reconstructs the scene in simulation, writes an expert controller, and writes a success judge. Then the simulator runs that expert from many initial states to generate exact trajectories. A video model, conditioned on real frames, renders those trajectories into photographic video. Importantly, all camera views are generated together so they stay consistent with one another and with the simulated motion. The final episodes include the rendered observations, the simulator actions, simulator states, contact events, task labels, and admission evidence. A major concern, of course, is trust. If the same agent writes the expert and the judge, it might approve its own mistakes. And if the renderer is asked to make things look real, it could hallucinate objects or drift away from the actual motion. So DataFarm is built around certification as a separate step. The expert is checked against a judge it cannot inspect, and it is only admitted on held-out initial states. The renderer is allowed to change appearance, but not geometry, object identity, or timing. Every rendered episode is then checked against the simulator state by machine checks and by an independent reviewer before it enters the dataset. In other words, production and certification are deliberately separated. The paper frames the whole system as a kind of compiler pipeline. There is a shared capability layer that contains the agent harness, roles, templates, and verifiers. That shared layer is fixed across tasks. For each new request, the agent produces task-specific artifacts: a grounding record, a task contract, a scene and initial-state sampler, an expert controller, a judge specification, a render contract, and batch and dataset configurations. This separation is important because it means the farm does not depend on a hand-built adapter for each new task. Instead, the agent generates the missing pieces itself, while the shared harness stays the same. The authors also analyze cost explicitly. The farm has a one-time adaptation cost for grounding, simulator qualification, and expert compilation, plus a render contract cost. After that, each admitted episode has a marginal cost that comes from simulation, rendering, and review, divided by the admission yield. The point is that failed attempts are not free, but the expensive parts of teleoperation are gone: no robot time, no operator time, no physical resets, and no hardware risk. In their reported numbers, adapting a task costs about $21 in API usage on average, plus a $5 render contract, and each further episode costs about $0.80 on average. They report that one hour of synchronized three-view data costs about $107 on average, which is slightly below a teleoperation quote from China and well below labor cost alone in North America. So does it actually work? They evaluate on five real-robot tasks spanning rigid, articulated, deformable, and display-bearing objects. That variety matters because it tests whether the farm can handle not just easy rigid scenes, but also things like cloth and screens that are notoriously awkward for pure reconstruction pipelines. They train policies directly on farm data and deploy them on the real robot. The main result is that policies trained only on farm data match policies trained on the same number of teleoperated episodes in the original scene. The paper reports 93 successes versus 84 successes in 100 trials, which indicates that the farm is not merely producing pretty videos; it is producing data useful enough to train a working policy. Even more interestingly, the farm-trained policies generalize much better when the appearance changes. Under an unseen tablecloth and colored light that were absent from both training sets, the farm-trained policies achieve 87 successes, while the teleoperation-trained policies drop to 46. The authors attribute this to the appearance variation introduced by the farm around fixed, exact trajectories. In other words, the simulator keeps the task structure precise, while the video model broadens the visual support. They also compare against a conventional real-to-sim-to-real baseline with domain randomization. That baseline performs much worse, with only 26 and 21 successes in the reported settings. This is an important result because it suggests that the gain is not just from using synthetic data in general. The particular combination matters: exact simulator actions plus realistic, real-image-conditioned rendering plus careful admission checks. The paper also checks whether the rendered data really looks and behaves like real camera recordings. Image statistics place farm episodes more than twice as close to real camera frames as the simulator’s own renderings. At the same time, the farm’s renderer keeps all views faithful to the simulated motion, whereas simply calling the video model directly does not. That distinction is crucial: realism alone is not enough if the motion is inconsistent or the action labels are wrong. There are still limitations to keep in mind. The system depends on one calibration episode, so it is not magic; it still needs a real scene recording to ground the task. The paper also makes clear that the agent’s output must be carefully certified, because language models are unreliable judges of their own work. And although the renderer can vary appearance, the method is still constrained by what the simulator can represent and what the admission checks can verify. So this is not a replacement for all forms of robot data collection. It is a way to automate a large and expensive part of it, especially when the task can be compiled from a short real-world calibration and when appearance variation is more important than full physical reconstruction. The broader takeaway is that the paper is not just proposing a new rendering trick. It is proposing an operational model for robot data collection: let an agent do the engineering, let simulation provide exact control, let a video model provide realistic observation, and let independent checks certify the result. If that pipeline holds up more broadly, it could change how robot datasets are produced for new tasks and new scenes.