Daily AI Paper

When people talk about evaluating AI agents, they usually mean something fairly clean: here is the task, here are the requirements, solve it. But that is not what real customer delivery looks like. In practice, a forward-deployed engineer is dropped into a messy organization, has to figure out what the real problem is, has to ask the right people the right questions, has to work inside live systems, and often has to discover that the sponsor’s original request is not the thing that actually matters. This paper is about that last mile of AI delivery, and it argues that this part of the job has been largely missing from existing benchmarks. The motivating example is very concrete. A vice president at a drilling operator wants a flashy generative-AI success story for a board meeting. The sponsor asks for a daily summary email and an “Ask your wells” chat box, and says the downtime-coding system is out of scope. A model could build exactly that, pass every functional check, and still fail to improve the customer’s business metric at all. Why? Because the real bottleneck is elsewhere: the downtime-coding script is misclassifying some downtime, which causes disputed monthly credit claims and suspends the credit for that month. Fixing that means revisiting the supposedly out-of-scope system, checking the contract rule, confirming the interpretation with a cost engineer, and getting approval from someone who is unavailable that week. That is the kind of problem FDE-Gym is designed to capture. The broader context is that most benchmarks do not test this combination of skills. Software engineering benchmarks often give issue reports and hidden tests. Deployment benchmarks give specifications. Enterprise benchmarks give predefined tasks. Clarification benchmarks hide information, but the underlying task is still fixed. Some newer benchmarks get closer to reality, including ones that evaluate customer-service systems, agent delivery roles, or misleading user requests. But they still tend to isolate pieces of the job. FDE-Gym tries to combine them: unclear or even misdirected sponsor requests, hidden constraints that must be discovered, live stakeholder interaction, and evaluation based on whether the delivered system actually improves the customer’s own outcome. The benchmark contains 95 tasks drawn from 20 customer scenarios across 18 sectors. These scenarios were adapted by industry experts from real-world delivery projects carried out in China, Singapore, and the United States since 2024. Each scenario includes an original task and several variants, so the benchmark is not just a set of one-off examples. The variants matter because they change key facts: which rule is outdated, who knows the crucial information, or whether the sponsor’s request is actually aimed at the right problem. That makes it harder for models to memorize a pattern and easier to see whether they can reason from the evidence in front of them. Each task represents a customer engagement with three parts: the customer organization, the sponsor who assigns the task, and one production system to improve. The agent starts in a workspace that includes the task instruction, a copy of the production system, relevant documents, historical data, and a staff directory. It can use Python and SQLite, but cannot install extra packages or browse the general internet. If it finishes, whatever is in the system directory is deployed to production without modification, and the production system continues to call the original entry points. So the agent has to integrate its changes in a way that actually takes effect. A key design choice is that the agent is not told some of the most important meta-information. It is not explicitly informed that stakeholder contact costs money, that some people are irrelevant or should not be contacted, or that the sponsor’s request may be misdirected. Instead, it has to infer that from the documents, data, and staff directory, which is much closer to how an FDE works in practice. The benchmark also uses scripted stakeholder simulations for reproducibility. More than 130 stakeholders are simulated across the 20 scenarios, with fixed knowledge, misconceptions, and disclosure conditions. That matters because user-simulation models can shift measured success rates, so the authors wanted a stable environment. The paper evaluates performance with a score that is meant to reflect real delivery value, not just task completion. The main quantity is normalized business uplift. In plain language, they compare the customer’s business metric for three versions of the system: the unchanged baseline, the delivered system, and an expert-authored reference solution. A score of 0 means no improvement over the baseline, 1 means matching the reference solution, and values below zero mean the delivered system made things worse. If an operational incident occurs, such as an unauthorized action, positive uplift is set to zero while negative uplift is preserved. Then they subtract stakeholder contact cost, which includes chargeable messages, trial runs, and even waiting time in some scenarios. So the final FDE-Gym score rewards real improvement but penalizes wasteful interaction and risky behavior. They also define a task as resolved only if the score is at least 0.8 and there is no incident. That is a high bar, and intentionally so. It means the system is not just somewhat helpful; it is close to the expert reference and safe enough to count as a success. To study the role of missing information, the authors create four instruction levels. At L0, the task explicitly states the underlying problem, the relevant rules, and the scoring criteria. At L1, the rules and constraints are withheld. At L2, the problem itself is only vaguely described. At L3, the agent gets the sponsor’s original request, which may not even target the real issue. L3 is the main setting because it most resembles the starting point of a real delivery project. The important point is that the environment, stakeholders, and grading stay the same across all four levels; only the instruction changes. So differences in performance isolate the effect of information withheld. The authors evaluate 13 model configurations from seven providers. The headline result is that current models are still far from the reference solution in this setting. In the realistic L3 setting, the best configuration, Claude Opus 5.5, achieves a mean score of 0.56 and fully resolves only 14 of the 95 tasks. Five of the thirteen configurations resolve none. If you let yourself pick the best-performing configuration separately for each task, the score rises to 0.70 and 30 tasks are resolved, but that still leaves a substantial gap. One of the most important findings is that discovering unstated constraints is the primary bottleneck. When the relevant rules and constraints are provided, the three leading models score between 0.73 and 0.82. When they have to uncover those constraints themselves, their scores drop sharply, into the 0.39 to 0.62 range. That tells us the main difficulty is not simply implementation. It is figuring out what problem should be solved in the first place, and what hidden rules govern the solution. But the paper also shows that information discovery alone does not solve the problem. At L3, the leading models are able to obtain 72 to 85 percent of the key facts available from stakeholders. Yet the delivered systems only reach about 0.60 to 0.70 of the reference improvement. In other words, even when the models ask many of the right questions and learn most of the important facts, they still do not consistently translate that information into an effective deployed solution. Stakeholder contact cost lowers the final score further by 0.14 to 0.22, so interaction itself is not free. The results also expose failure modes that are easy to overlook if you only look at average scores. Weak models can actually make things worse than doing nothing. Claude Haiku 4.5 falls below the do-nothing baseline on 79 of 95 tasks, and Mistral Small 4 falls below it on 51 tasks even before contact costs are deducted. That is a strong warning: in a delivery setting, a model that looks merely “imperfect” can still harm the customer. Another striking result is that no model is robust across all scenarios. In five of the 20 scenarios, no model reaches a mean score of 0.5. And for a leading model, the median score range across tasks within a scenario is still quite wide, around 0.34 to 0.44. So performance is not just low on average; it is uneven and scenario-dependent. What should we take away from this? First, there is a real gap between solving well-specified benchmark tasks and delivering useful AI systems for real customers. Second, the hardest part is often not coding, but identifying the underlying problem and the hidden constraints that determine whether a change actually helps. Third, even when models gather most of the relevant information, they still struggle to turn that into a working, business-improving deployment. The paper’s limitations are also important. FDE-Gym is still a benchmark, so it simplifies reality in order to measure it. Stakeholders are scripted, production systems are simulated, and the scenarios, while grounded in real delivery projects, are still curated tasks. The scoring system is carefully designed, but any single metric will miss some aspects of real-world judgment, trust, and organizational politics. And because the benchmark uses expert-authored reference solutions, it measures closeness to those solutions rather than every possible good answer. Still, the contribution is substantial. FDE-Gym gives the field a way to test the part of AI delivery that happens after the prompt and before the happy path: the messy, human, operational work of discovering what really matters and making it work in a live environment. That is the last mile, and this paper shows that today’s models are not there yet.