DeepSeek Controls Robots Without Vision Using Tool Use

Overview

A new development in robotic control is challenging the long-held assumption that artificial intelligence models require sophisticated visual systems to manipulate physical objects. Researchers have demonstrated that DeepSeek, a powerful language model, can effectively control robots without processing a single pixel of visual data. This counterintuitive finding emerges from rigorous testing on the SEAR benchmark, where the model not only kept pace with leading vision-embedded systems like Gemini but also outperformed them on the most complex task families.

The breakthrough suggests that the field’s heavy investment in multi-modal vision may be overstated for certain robotic applications. Instead, the critical component appears to be agentic reasoning and the ability to leverage external tools intelligently. By decoupling decision-making from direct sensory input, this approach offers a streamlined path toward more robust and scalable robot autonomy.

The Tool-Use Architecture

The core of this achievement lies in how DeepSeek interacts with its environment through specialized software interfaces rather than raw camera feeds. The model does not "see" the world in the traditional sense; instead, it queries other algorithms to retrieve structured information it can act upon.

When tasked with moving an object, DeepSeek first calls SAM3, a segmentation model, to generate a precise mask identifying the named target. This mask defines the exact spatial boundaries of the object in question. However, a 2D image mask is insufficient for physical manipulation, so the system then processes the mask alongside depth images to calculate accurate 3D coordinates.

This two-step process transforms visual data into actionable geometric instructions without requiring the language model itself to perform computer vision tasks. The architecture relies on a clear division of labor: the language model handles planning and logic, while specialized tools handle perception and coordinate transformation.

Benchmark Performance and Capabilities

To validate this approach, researchers subjected DeepSeek to the SEAR benchmark, a standardized suite designed to test embodied AI agents in realistic robotic scenarios. The results were striking, particularly when compared against Gemini, a model renowned for its strong multi-modal capabilities.

• Competitive Parity: DeepSeek maintained performance levels comparable to Gemini across the majority of standard robotic tasks.
• Superior Complexity Handling: On the hardest task families within the benchmark, DeepSeek surpassed Gemini, suggesting that its tool-use methodology scales better under complex conditions.
• Zero-Pixel Input: Despite the absence of visual input, the model successfully executed nuanced manipulation commands, proving that visual awareness is not a prerequisite for functional robotic control.

These findings indicate that while vision models are powerful, they are not always the optimal solution for every layer of the robotic stack. The language model’s ability to reason about tool outputs proved more advantageous in difficult scenarios than raw visual processing.

Implications for Agentic Reasoning

This research underscores a shifting perspective in artificial intelligence development. The industry has largely operated under the belief that adding more sensory modalities, particularly vision, directly correlates to better performance. However, this study highlights that agentic reasoning—the capacity to plan, sequence actions, and utilize tools—is often the determining factor in success.

"While vision helps with error assessment, agentic reasoning and tool use are what matter most."

The role of vision is redefined in this context. It is not discarded entirely but is relegated to a supportive function. Visual data can assist in error assessment and fine-tuning, but the primary driver of successful task completion is the model's logical framework for interacting with the world. By abstracting away the need for the LLM to interpret pixels, the system becomes more modular and potentially more reliable.

Future Directions in Robotics

The ability to control robots without vision introduces several compelling advantages for future applications. First, it reduces computational overhead, as the heavy lifting of perception is offloaded to specialized, efficient models like SAM3. Second, it enhances modularity; a single language model can potentially control various robotic platforms by simply changing the toolset it queries.

This architecture also opens doors for scenarios where visual data is unreliable or unavailable. In low-light environments, obscured views, or settings with significant visual noise, a vision-dependent system might fail, whereas a tool-using language model could still succeed by relying on depth sensors or other non-visual data sources.

As research progresses, the line between perception and action continues to blur. The success of DeepSeek in this domain suggests that the next wave of robotic intelligence will prioritize smart reasoning and tool integration over brute-force sensory processing. This paradigm shift could accelerate the deployment of autonomous robots in diverse, unstructured environments.