Deliveroo Cuts AI Churn Alerts by 90% in Retention Overhaul

Overview

Deliveroo has dramatically reduced the volume of churn alerts it sends to its retention teams, cutting the queue by more than 90%. The move addresses a critical operational failure: the previous AI model flagged so many restaurants as at-risk that account managers were overwhelmed and unable to prioritize effectively.

By shifting from a high-recall approach to one constrained by human capacity, the company aims to ensure that every outreach effort targets a genuinely vulnerable partner. However, while the statistical improvements are significant, Deliveroo acknowledges that the ultimate test lies in whether these sharper interventions actually prevent restaurants from leaving the platform.

The overhaul, detailed in a first-party engineering account published on 22 September 2026, highlights a growing trend in growth marketing where AI systems are being reined in not by accuracy alone, but by the scarce resource of human attention.

"Deliveroo’s churn model cried wolf. Now a randomised holdout will show whether the sharper list works."

The Cost of False Positives

For marketplaces like Deliveroo, which connect consumers, riders, and partners, restaurant retention is economically vital. The platform invests significant time and capital in acquiring and onboarding each partner. When a restaurant churns, it not only represents a lost revenue stream but also shrinks the selection available to customers, potentially degrading the user experience.

The previous churn prediction system was engineered for high recall, meaning it prioritized catching as many potential leavers as possible. In a binary classification context, this minimizes false negatives—the risk of missing a restaurant that was about to leave. However, because actual churn is a rare event, optimizing strictly for recall resulted in a flood of false positives.

Nearly every account began to look urgent to the algorithm. This created a paradox where the retention system itself became a source of churn risk. Account managers faced an unmanageable queue and could not differentiate between a restaurant with a minor dip in performance and one on the verge of closing.

The opportunity cost of these false alarms was substantial. Every call made to a healthy restaurant was a call not made to a genuinely vulnerable partner. As Firat Yaman, Deliveroo’s data scientist and economist, noted, the old system optimized for the wrong scarcity. It treated the problem as a data classification challenge rather than a resource allocation problem.

Rebuilding Around Scarce Resources

The new system flips the objective function. Instead of asking how many future churners the model can identify, Deliveroo now asks how many interventions its team can realistically execute. The replacement model estimates a probability for every restaurant, ranks the accounts by risk, and applies an adjustable threshold based on weekly outreach capacity.

This approach introduces a direct link between machine learning outputs and operational constraints. By raising the threshold for what constitutes a "high-risk" alert, the company ensures that only the most likely candidates enter the queue. The trade-off is a reduction in recall, meaning some restaurants that would have been caught by the old system are now ignored. Deliveroo argues this is an acceptable loss given the previous system’s inability to act on its own warnings.

To maintain utility, the pipeline incorporates Shapley values, a game-theoretic method used for feature attribution. This allows the system to explain why a specific restaurant received a high churn score by highlighting which input features contributed most to the prediction. For an account manager preparing for a call, understanding that a drop in order volume or a change in cuisine category drove the risk score can inform the conversation strategy.

The model runs weekly via an Argo pipeline, ensuring that risk scores and supporting data are delivered to internal users automatically. However, the article notes it remains unclear whether providing these explanations actually improves the quality of outreach or if it merely adds complexity for the end-user.

Metric	Old System	New System
Optimization Goal	High Recall	High Precision / Capacity-Constrained
Alert Volume	High (Unmanageable)	Reduced by >90%
Primary Risk	False Positives	False Negatives
Interpretability	Limited	Shapley Values

Validation and Experimentation

Predicting future churn is fraught with the danger of data leakage, where information from the future inadvertently influences the training or validation process. To mitigate this, Deliveroo employed rolling window cross-validation. Configurations were trained on historical observations and validated on subsequent periods, ensuring that the model’s performance metrics reflected true predictive power rather than temporal overlap.

Offline testing results showed an eightfold improvement in precision alongside the 90% reduction in flagged restaurants. Deliveroo did not disclose absolute precision figures, sample sizes, or confidence intervals, leaving some ambiguity about the baseline performance. An eightfold lift could stem from a very low initial precision, and fewer alerts can appear efficient even if the remaining interventions have no real-world impact.

Consequently, the offline metrics are only part of the story. The company is now conducting a randomized holdout experiment to measure the true business impact. Restaurants flagged by the new model are split into treatment and control groups. The retention team receives alerts for the treatment group, while alerts for the control group remain hidden.

"That comparison tests the combined pipeline. Prediction, prioritisation, human follow-up and the action offered to a partner all sit inside it."

This A/B test is crucial because it evaluates the entire system, not just the algorithm. It isolates whether the combination of better targeting and human intervention leads to statistically lower churn rates. If the treatment group does not show significantly higher retention, the efficiency gains from the AI overhaul may be illusory, suggesting that the problem lies in the outreach strategy rather than the identification of at-risk partners.