New AI Tool Lets Agents Point at Your Screen Instead of Blocking Work

Overview

A new open-source macOS utility called bigarrow is addressing a persistent friction point in the evolution of autonomous AI agents: the moment when an agent must ask a human to perform a physical action. While modern AI systems can refactor code, write migrations, and explain complex concepts, they frequently hit a wall when an interaction requires human authorization or presence on the local machine.

The tool provides a simple solution: it draws large, animated arrows, boxes, and text directly onto the user’s screen, pointing precisely at the element that requires attention. By allowing agents to "point" rather than attempt to click through barriers, bigarrow aims to streamline the handoff between automated workflows and human oversight without disrupting the user’s current focus.

The Human-in-the-Loop Problem

As software agents become more capable, they are increasingly tasked with end-to-end workflows that were previously the domain of human operators. However, operating systems like macOS enforce strict security boundaries that prevent software from interacting with certain permission prompts, OAuth consent screens, or two-factor authentication dialogs.

Traditionally, when an agent encountered such a barrier, it would output a message to the terminal, such as "please click Allow in the dialog." This approach is inefficient because it requires the user to constantly monitor the terminal output rather than their actual workflow. It creates a disjointed experience where the agent is working in the background while the user is left guessing which window or dialog requires attention.

"Your AI agent can refactor a monorepo... but when it needs you to click one button it prints 'please click Allow' into a terminal you are not looking at."

Bigarrow solves this by giving the agent a visual "finger." Instead of shouting into the void of a command line, the tool overlays a clear, unmissable visual cue on the desktop, guiding the user’s eye directly to the required interaction point.

How Bigarrow Works

The utility is built as a lightweight Swift binary that operates without daemons, menu-bar icons, or accounts. Notably, the developers emphasize that there is no AI logic inside the arrow tool itself; it is purely a visualization engine. This design choice ensures minimal resource usage and eliminates privacy concerns regarding telemetry or background processing.

The tool utilizes macOS accessibility features to render graphics that sit above all other windows. These overlays are click-through, meaning they do not intercept mouse events or steal focus from the application the user is currently working in. Once the user interacts with the target element, the arrow can automatically disappear.

Key capabilities include:

• Element-based pointing: Agents can target specific UI labels, such as pointing at an "Allow" button in System Settings.
• Coordinate-based pointing: For situations where UI labels are insufficient, agents can draw directly at specific screen coordinates.
• Multi-display support: The tool functions correctly across all connected displays and virtual Spaces.
• Auditory alerts: A --say flag allows the system to read the sign aloud, effectively calling the user back to their desk if they are away from the machine.

Use Cases and Applications

While the primary use case is guiding AI agents, the tool is versatile enough for several other scenarios:

1. Remote Assistance: Tech support or collaborative debugging can be simplified by pointing at specific gear icons or windows rather than attempting lengthy verbal descriptions.
2. Guided Onboarding: Administrators can create step-by-step tours for complex settings panes, waiting for human confirmation before proceeding to the next step.
3. Documentation and Demos: The tool can render arrows directly into PNG files for static documentation or highlight key areas during screencasts.
4. Debugging Coordinates: Developers can verify accessibility coordinates by pointing at elements and reviewing the output in JSON format without actually drawing the arrow.

Installation and Technical Details

Bigarrow is distributed under the MIT license and is available via Homebrew. Installation is straightforward, with a dedicated command to integrate the tool as a skill for popular AI coding assistants like Claude Code and Codex.

brew install franzenzenhofer/tap/bigarrow
bigarrow install-skill

For those preferring to build from source, the project requires Xcode 16 or newer and macOS 14 or newer. The resulting binary is efficient and includes comprehensive flags for customization, allowing users to adjust shape (straight, bend, or zigzag), style (arrow, ring, or box), color, and size.

The tool provides several exit codes to facilitate robust scripting: 0 for success, 2 for bad input, 3 if the target is not found, and 4 for missing permissions. This structure allows agents to handle errors gracefully and inform users precisely why a visual cue could not be generated.

Conclusion

As the line between autonomous software and human-operated systems continues to blur, tools that facilitate clear communication between the two are becoming essential. Bigarrow represents a modest but effective step toward smoother human-agent collaboration. By replacing ambiguous terminal messages with precise visual guidance, it reduces the cognitive load on users and allows AI agents to operate more fluidly within complex, permission-heavy environments.

The project’s minimalistic philosophy—no accounts, no telemetry, no AI inside—positions it as a trust-worthy utility for privacy-conscious developers and organizations integrating AI into their macOS workflows.