Daily AI Paper

When people talk about benchmarking AI agents, they often mean tasks that are already neatly defined: here is the bug, here is the spec, here are the hidden tests, go solve it. But real customer delivery is messier than that. In practice, a forward-deployed engineer has to figure out whether the sponsor is even asking for the right thing, discover unstated constraints, work inside live systems, and make sure the result actually improves the customer’s business outcome. This paper is about that gap between solving a well-posed task and delivering something that truly helps a real customer. The authors motivate the problem with a concrete example from a drilling operator. A vice president wants a daily summary email and an “Ask your wells” chat box for a board meeting. On paper, that request sounds straightforward, and a team could build exactly that and pass all functional checks. But if the real business bottleneck is a downtime-claim process that is incorrectly attributing costs, then the flashy new interface does nothing for the customer’s actual metric. Worse, the sponsor may have explicitly said that the downtime-coding system is out of scope, so a literal implementation of the request misses the point entirely. The engineer has to notice that the stated request is misdirected, identify the real underlying issue, confirm the relevant rule with the right stakeholder, and get approval to change something that was supposedly excluded. That is the central idea behind FDE-Gym, a benchmark for the “last mile” of AI delivery. The benchmark is built to capture the full workflow of customer-facing engineering rather than just implementation. It includes 95 end-to-end tasks drawn from 20 customer scenarios across 18 sectors, and the scenarios were adapted by industry experts from real delivery projects. The important part is that these are not just synthetic toy problems. They are designed to resemble the kinds of ambiguous, constraint-heavy engagements that happen in enterprise work. To understand the contribution, it helps to place it in the broader benchmark landscape. Classic software-engineering benchmarks usually give agents an issue report or a hidden test suite. Enterprise or deployment benchmarks often provide a clear specification. Clarification benchmarks hide some information, but the underlying task is still fixed. More recent work has moved closer to real delivery, including benchmarks where agents build customer-service systems or play the role of a forward-deployed engineer. But the authors argue that those settings still isolate only parts of the problem. FDE-Gym tries to combine the hard parts in one place: unclear requests, stakeholder interaction, production-like systems, and scoring based on the customer’s own business metric. The benchmark environment is designed to feel like a real engagement. The agent starts in a workspace with the sponsor’s task instruction, a copy of the customer’s production system, relevant documents and historical data, and a staff directory with contact details. The agent can use standard Python and SQLite, but cannot install extra packages or browse the general internet. If the agent sends network requests, they can only go through the company’s LLM gateway, which is metered. When the agent finishes, whatever is in the system directory gets deployed to production without modification, so the solution has to actually fit the existing entry points. That detail matters because in real delivery, you do not get to redesign the entire environment from scratch. A major part of the benchmark is that the agent is not told everything up front. It is not explicitly informed that stakeholder contact costs money, that some people are irrelevant or should not be contacted, or that the sponsor’s request may be pointing at the wrong problem. Those are exactly the kinds of things a real FDE has to infer. The clues are there, but they are distributed across documents, code, data, and the staff directory. The benchmark also assumes that some documents are outdated and some legacy code reflects incorrect assumptions, which again mirrors real work. One clever design choice is that every scenario appears in multiple instruction levels. At level L0, the task explicitly states the underlying problem, the relevant rules, and the scoring criteria. At L1, the rules and constraints are removed. At L2, even the description of the problem becomes vague. At L3, the agent only sees the sponsor’s original request, which may not even address the true issue. The environment and grader stay the same across levels, so the only thing that changes is what information the agent starts with. That lets the authors isolate the cost of missing facts and unclear problem framing. They use L3 as the main setting because it most closely resembles a real customer engagement at the start. Now, how is success measured? Not by whether the agent wrote code that looks plausible, but by whether the delivered system improves the customer’s own business metric on held-out production data. The paper defines a normalized business uplift score. The unchanged system is the do-nothing baseline and gets zero. An expert-authored reference solution gets one. If the delivered system is worse than baseline, the score can go below zero. There is also an incident adjustment: if a predefined operational incident happens, positive uplift is wiped out, though negative uplift remains. On top of that, stakeholder interaction has a cost. Every chargeable message, trial run, or waiting period can subtract from the final score. So an agent that gets the right answer only after excessive back-and-forth may still perform poorly. The single-run FDE-Gym score is the business uplift minus contact cost, and a task is considered resolved only if the score is at least 0.8 and there is no incident. The benchmark construction is also intended to prevent shortcut learning. Each scenario comes from a real delivery project, but the authors create between two and seven variants per scenario, changing things like which rule is outdated, who has the key information, and whether the sponsor’s request is actually the right problem. That means a solution that works for one version may fail on another. They also simulate more than 130 stakeholders with scripted responses for reproducibility, so the same question always leads to the same kind of answer. This is important because if the simulation itself were noisy, it would be hard to tell whether a model succeeded because it reasoned well or just got lucky. The evaluation covers 13 model configurations from seven providers. The headline result is that even the best model is far from the reference solution in the realistic setting. At the most realistic level, L3, Claude Opus 5.5 achieves the best mean score, 0.56, and resolves only 14 of the 95 tasks. Five of the 13 configurations resolve none at all. If you imagine picking the best model separately for each task, performance improves, but still only reaches a mean score of 0.70 and resolves 30 tasks. So even with oracle model selection, there is still a substantial gap. The most important bottleneck is discovering unstated constraints. When the relevant rules and constraints are given directly, the three leading models score in the 0.73 to 0.82 range. But when they have to uncover those constraints themselves, scores fall sharply into the 0.39 to 0.62 range. Interestingly, moving from the fully specified setting to the real-world style setting does not always cause a huge drop in the task instructions themselves; the paper notes that L3 scores are at most 0.08 points lower than L1 scores on the original tasks. That suggests that the hard part is not just reading a vague request. It is figuring out the hidden operational reality behind it. The authors also show that collecting information is not enough. At L3, the leading models obtain 72 to 85 percent of the key facts available from stakeholders, but their delivered systems still reach only 0.60 to 0.70 of the reference improvement. In other words, even when the model asks around and learns a lot, it may still fail to translate that knowledge into the right system change. And once contact costs are deducted, scores fall further by 0.14 to 0.22. So there is a second gap after information gathering: implementation and judgment. There are also some sobering failure modes. Weak models can make things worse than doing nothing. Claude Haiku 4.5 falls below the baseline on 79 of 95 tasks, and Mistral Small 4 falls below baseline on 51 tasks even before contact costs are counted. That means a poorly chosen or poorly executed delivery can actively harm the customer’s metric. Another striking result is that no model performs well across all scenarios. In five of the 20 scenarios, no model even reaches a mean score of 0.5. And within a scenario, the spread in scores for a leading model can still be large, which suggests that these tasks are not just difficult in general; they are unevenly difficult depending on the customer context. What should we take from all this? First, benchmark results on well-specified tasks are not enough to tell us how useful an agent will be in a real customer environment. Second, the biggest challenge is often not coding, but problem discovery: understanding what actually matters, who to ask, and which constraints are real. Third, even when an agent learns the right facts, it still has to convert them into a system that improves the business metric without causing incidents or wasting stakeholder time. As for limitations, the benchmark is still a simulation. Stakeholders are scripted, production systems are controlled, and the scenarios, while grounded in real projects, are adapted into a benchmark format. That means FDE-Gym captures a lot of realism, but not the full unpredictability of live enterprise work. The scoring also focuses on measurable business uplift, which is appropriate for delivery tasks, but real deployments can involve additional concerns like long-term maintainability, trust, and organizational politics. And because the benchmark uses a fixed set of scenarios, there is always the question of how well performance will generalize to new domains and new kinds of ambiguity. Even so, FDE-Gym makes a strong point: there is a substantial gap between an AI system that can satisfy a request and one that can deliver value for a real customer. Closing that gap will likely require better constraint discovery, better stakeholder reasoning, and better end-to-end planning under real operational costs. That is the last mile this paper wants the field to pay attention to.