AI Agents Struggle with Real-World Customer Delivery

Overview

A new benchmark study reveals a significant gap between how AI agents perform in controlled environments and their ability to deliver effective solutions in real-world customer settings. The research, titled FDE-Gym: The Last Mile of AI Delivery, introduces a rigorous testing framework designed to evaluate forward-deployed engineers (FDEs) and AI systems under conditions that mirror actual industry challenges.

Published on arXiv in October 2026 by Dongxu Yang and 25 co-authors, the paper argues that existing benchmarks are too well-specified to reflect the messiness of live customer delivery. Real-world scenarios require agents to navigate ambiguous requirements, interact with stakeholders, and uncover unstated constraints—skills that even the most advanced models currently struggle to master.

"Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different."

The FDE-Gym Benchmark

FDE-Gym was constructed to test AI agents on 95 end-to-end delivery tasks drawn from 18 different sectors. These tasks were adapted by industry experts from actual delivery projects, ensuring they reflect the complexity and variability of real customer interactions.

Simulated Customer Environments

The benchmark utilizes simulated customer environments where agents interact with scripted stakeholders and operate on production-like systems that can be modified. This setup allows researchers to measure not just whether an agent can write code or generate text, but whether it can deliver a working solution that improves a customer’s specific success metrics.

Scoring is based on two primary factors:

1. Business Uplift: The improvement in the customer’s own success metric, normalized between 0 (no change) and 1 (reference solution).
2. Interaction Penalties: Points are deducted for excessive messages, trial runs, and waiting time, reflecting the efficiency costs of real-world deployments.

An online platform accompanying the benchmark also allows human experts to practice FDE skills under the same grading criteria, providing a baseline for comparison against AI performance.

Key Findings: The Gap Between Theory and Practice

The study evaluated 13 model configurations from seven different providers, and the results highlighted a substantial disconnect between solving well-defined problems and delivering effective AI systems.

In the realistic setting, where unstated constraints must be discovered without explicit guidance, the best-performing configuration achieved a mean FDE-Gym score of only 0.56. More starkly, it resolved just 14 of the 95 tasks to a high standard (a score of at least 0.8 with no incidents).

Discovering Unstated Constraints

The primary bottleneck identified is the difficulty agents face in discovering unstated constraints. Performance drops sharply when models are required to infer implicit needs or limitations that are not explicitly mentioned in the task description.

However, the study found that information discovery alone does not fully explain the performance gap. Even on the 23 tasks where the best configuration successfully obtained every key fact from stakeholders, it reached a normalized business uplift of 0.8 or above on only 10 of those tasks. This suggests that once requirements are clear, models still struggle with implementation, integration, and delivering robust, working solutions.

These results indicate that the current generation of AI agents, while proficient at isolated coding or reasoning tasks, lacks the holistic operational competence required for effective customer delivery.

Implications for AI Development

The findings from FDE-Gym underscore the need for new evaluation standards in artificial intelligence. As AI systems are increasingly deployed in production environments, benchmarks must evolve to test not just accuracy, but also adaptability, stakeholder communication, and operational efficiency.

The research suggests that future work should focus on:

• Improving agent capabilities in ambiguous and open-ended environments.
• Developing better methods for constraint discovery and inference.
• Creating training frameworks that emphasize end-to-end delivery outcomes rather than isolated task completion.

Human-AI Collaboration

The availability of an online platform for human practice raises interesting questions about the role of human forward-deployed engineers in an AI-augmented future. While the gap is wide, the structured nature of the benchmark allows for targeted skill development, potentially helping humans and AI agents collaborate more effectively in real-world delivery scenarios.

Conclusion

As AI continues to integrate into critical business operations, the lessons from FDE-Gym are crucial. The "last mile" of AI delivery—translating technical capability into tangible customer value—remains a formidable challenge. Until models can reliably handle ambiguity, discover hidden constraints, and deliver robust solutions in live systems, the promise of fully autonomous AI deployment will remain partially unfulfilled.