SealGate's Mobile Tunnel Lets AI Pilot Drones via Android

Overview

SealGate has introduced Mobile Tunnel, an open-source Android companion for its SealGate Gateway, enabling AI agents to observe and interact with Android devices remotely. This development allows agents to control not only the phone itself but also any hardware the device can reach via Wi-Fi, Bluetooth, USB, or NFC.

The technology was demonstrated through a significant experiment where an AI agent was given autonomous control of a DJI Neo drone via a standard Android smartphone. Tasked with finding and landing on a table from a single high-level prompt, the agent succeeded with minimal tuning. The project, led by Ilia Manolov and Eito Miyamura, represents a natural evolution from previous stdio tunnel implementations that allowed cloud-based AI to securely interact with local computer files and applications.

From Computer Control to Physical Devices

For some time, SealGate’s gateway has supported remote Model Context Protocol (MCP) servers, allowing cloud agents to interact with local environments. The introduction of stdiod, a daemon providing secure routes to these servers, expanded capabilities to include desktop applications, development environments, and local file systems. However, computers represent only one category of connected hardware.

The team identified a gap in controlling physical devices that do not expose public APIs but are managed through proprietary applications or controllers. A DJI Neo drone served as the ideal test case because it combines several complex challenges: it requires continual visual feedback, actions must be composed over time rather than issued as single commands, and it normally demands that a human learn a specific control interface.

"In theory, instead of teaching every user how to pilot a particular device, we could teach an agent once and let the user describe the outcome they want."

This shift suggests a future where modern multimodal models can interpret user intent directly. Instead of manual operation, a user might simply instruct an agent to perform reconnaissance, identify a specific object, and land autonomously. The core question driving this experiment was whether existing agent technologies were mature enough to execute these tasks reliably without extensive manual programming.

The Challenge of Proprietary Hardware

The initial approach involved using the DJI RC-N3 remote controller. This option offered significant advantages, including transmission ranges of several kilometres compared to the roughly 50-metre limit of direct smartphone control, as well as deterministic physical inputs and a direct video feed. The proposed architecture would have involved running a custom MCP server alongside stdiod on a computer host, translating MCP calls into controller inputs, and feeding the drone's camera imagery back to the multimodal agent.

However, this path was blocked by proprietary barriers. The USB protocol between the DJI RC-N3 and its host is closed, and the team could not locate a sufficiently complete open-source implementation to build upon. Attempts to reverse-engineer the protocol were abandoned after an inspection app revealed that the communication was heavily obfuscated. Converting a one-week experiment into a substantial reverse-engineering project was deemed impractical.

An alternative considered was having the AI agent manipulate the Android UI while the RC-N3 remained connected. This proved ineffective because the critical navigation inputs are handled directly by the physical hardware of the controller rather than being exposed as usable on-screen controls. Consequently, the team pivoted to a simpler, albeit more limited, control path.

Using the Android App as a Control Interface

The second option utilized the DJI Fly Android app, which allows the Neo to connect directly to a smartphone via Wi-Fi. This method exposes virtual joysticks on the screen, enabling the agent to interact with the device in the same manner as a human operator. The agent observes the screen, decides on an action, taps or swipes the virtual controls, observes the resulting video feed, and repeats the process.

While this approach sacrifices range and does not provide the deterministic precision of a physical controller, it eliminates the need to reverse-engineer proprietary communication protocols. For the purposes of this experiment, the 50-metre range was sufficient, particularly given that current multimodal agents process information slowly enough that the drone was unlikely to travel far during a single battery cycle. The primary requirement was that the phone remain awake and stationary on the DJI Fly piloting screen throughout the operation.

To facilitate this, the team developed Mobile Tunnel, an MVP Android implementation of the device-side functionality behind SealGate stdio tunnels. Once configured, the Android phone becomes another reachable device through the SealGate Gateway.

Capabilities and Implications

Mobile Tunnel transforms the Android device into a versatile interface for AI agents. Depending on the specific MCP server and hardware involved, the system supports several key interaction types:

• Observing the current state of the phone screen to provide visual context.
• Performing taps, swipes, and gestures to navigate interfaces.
• Typing using the virtual keyboard for text input.
• Communicating with external hardware over Bluetooth and USB connections.
• Running device-local work processes.

This architecture opens the door for AI agents to interact with a wide array of consumer electronics, from 3D printers and embedded systems to media players and smart home devices, without requiring native API integrations for each specific product. By treating the smartphone as a universal remote control for the physical world, SealGate’s approach bridges the gap between digital AI reasoning and analog hardware operation.

The successful demonstration of autonomous drone piloting validates the potential for this technology. It proves that with the right abstractions, AI agents can move beyond purely digital environments to perform complex, real-world tasks guided only by high-level natural language instructions. As the technology matures, it may redefine how users interact with smart devices, shifting the burden from learning complex interfaces to simply stating desired outcomes.