AI Workload Placement: Decision Engine Reveals Cloud Speed Advantage

Overview

A new open-source project is challenging the assumption that processing AI workloads locally is always the fastest option. The AI Workload Placement decision engine, developed by independent engineer Keerthana0309, provides a tool for determining whether a task should run on a device, edge server, or cloud infrastructure.

The project addresses a specific gap in the current AI ecosystem. While platforms like AWS Greengrass and Azure IoT Edge handle deployment and fleet management, they often require immediate commitment to specific SDKs or hardware. This decision layer aims to provide a fast, no-commitment recommendation before any technical integration begins.

Benchmark Findings

The project includes a benchmarking script that tests a common summarization task across three environments: local, simulated edge, and cloud. The results, published in October 2026, offer a counterintuitive perspective on performance tradeoffs.

Test Methodology

The benchmark tasked each environment with summarizing five short news-style paragraphs. The local environment used the llama3.2:1b model via Ollama on a laptop at full power. The edge-simulated environment ran the same model but restricted it to a single CPU thread to approximate the constraints of weaker hardware. The cloud environment utilized Claude Haiku 4.5 via AWS Bedrock.

Performance Results

The data revealed that the cloud solution consistently outperformed the local options in terms of latency. Key findings include:

• Local performance: Completed requests in approximately 2.9 to 5.3 seconds at zero financial cost.
• Edge-simulated performance: Required between 3.5 and 4.7 seconds per request, also at zero cost.
• Cloud performance: Delivered results in just 1.1 to 1.5 seconds, with a marginal cost of roughly $0.0001 to $0.00015 per request.

The researcher noted that these numbers contradict the common presumption that local execution inherently wins on latency. Instead, the primary differentiator appears to be privacy and offline capability, advantages that only local and edge deployments can provide.

"The real tradeoff in this data wasn't latency or cost — it was privacy and offline capability, which only local/edge can offer."

Project Architecture

The repository provides two primary interfaces for users to interact with the decision engine. The core logic is contained within decide.py, which functions as a command-line tool. Users answer five quick questions regarding task type, privacy requirements, latency tolerance, connectivity status, and budget to receive a placement recommendation.

For users preferring a graphical interface, app.py wraps the same decision logic in a Streamlit web application. This allows for easier demonstration and interaction without requiring command-line proficiency. Both tools rely on the benchmark data generated by benchmark.py to ground their recommendations in empirical evidence rather than theoretical estimates.

Limitations of Edge Simulation

It is important to note that the current edge results are derived from simulation rather than physical testing. The edge-simulated run limits CPU threads on a standard laptop to mimic constrained hardware. The author acknowledges that this does not fully replicate the performance of actual edge devices, such as a Raspberry Pi, which face additional constraints related to memory and power efficiency.

Physical testing on dedicated edge hardware is listed as a planned next step. The developer intends to retrieve a Raspberry Pi from previous research projects to conduct more realistic benchmarks.

Strategic Implications

This project represents a continuation of the developer’s earlier work in edge and cloud workload placement, originally conducted in 2017. Back then, the focus was on Raspberry Pi networking and device control. Today, leveraging hands-on experience with AWS and Bedrock, the goal is to apply similar rigorous analysis to the rapidly evolving landscape of AI inference.

The tool is designed for infrastructure engineers and data scientists who need to make quick architectural decisions. By quantifying the tradeoffs between speed, cost, and privacy, it offers a data-driven starting point for deployment strategies.

Future Development

The roadmap for the project includes several expansions to its scope and utility. The immediate priority is replacing the simulated edge data with benchmarks from real hardware. Additionally, the team plans to expand the range of supported models and tasks beyond simple text summarization.

A longer-term vision includes the development of a cost and latency estimator that can predict performance based solely on a workload description, even before any code is executed. This would further reduce the friction involved in selecting the right infrastructure for AI workloads.

As an early-stage exploration, the project invites feedback and critique from the community. It serves as a practical case study in the ongoing debate over where artificial intelligence should run, suggesting that the cloud may be a more competitive option for latency-sensitive tasks than previously assumed.