jevman: The Pac-Man Benchmark for AI Decision Models

Overview

A new open-source benchmark titled jevman is challenging the artificial intelligence community to test decision-making capabilities in a retro gaming environment. By pitting modern AI models against the classic arcade game Pac-Man, researchers and developers can now evaluate how well these systems handle real-time strategy, risk assessment, and spatial reasoning under strict latency constraints.

The benchmark, currently at version 1.13, invites participants to submit models that interface via a simple HTTP endpoint. Six models have already completed 100 games each against the original scripted ghosts, establishing an initial leaderboard. The project aims to provide a transparent, reproducible standard for measuring AI decision quality, distinct from the heavy computational demands of large-scale training benchmarks.

How the Benchmark Works

jevman is designed to be accessible to any model that can expose an API. The core premise involves a Pac-Man agent navigating a maze while avoiding four ghosts. At every junction, the game sends the current maze state, including the positions of pellets and ghosts, as a JSON object. The AI model must return a direction, optionally accompanied by probability distributions for each possible move.

The competition imposes tight deadlines to simulate real-world decision latency. Models have only two seconds to respond to each junction query. If an answer exceeds this time limit, the system replaces the AI's choice with a simple backup rule. These forced moves are tracked separately, providing insight into a model's efficiency as well as its strategic depth.

Each model plays 100 games against the classic ghost behaviors. A game ends when Pac-Man loses all three lives or after a five-minute cap, though no game has yet approached the time limit. The longest recorded session lasted two minutes and 24 seconds, indicating that even slower models can sustain engagement for significant periods.

Scoring and Methodology

Transparency is a cornerstone of the jevman project. Every game is recorded in its entirety, allowing for exact replays of any match. This feature ensures that results can be independently verified and debugged, a critical component for scientific rigor in AI evaluation. The leaderboard does not rely on opaque proprietary metrics but rather on mean scores with a 95% margin of error.

The ranking system uses statistical confidence intervals to determine ties. If two models' scores fall within each other's margin of error, they are ranked as tied. This approach prevents minor performance fluctuations from creating artificial hierarchies among similarly capable models. The high score reported for each model reflects its best single-game performance, offering a look at peak capability alongside average reliability.

Participants are expected to run their own benchmarks using the provided command-line tools. The repository includes a 34-line example endpoint to help developers get started quickly. Once a model is behind an endpoint, users can submit their results via a pull request to the public repository. Continuous integration systems then replay every submitted game to validate the score before adding the model to the leaderboard.

The benchmark measures more than just raw intelligence; it evaluates how effectively an AI can process spatial data and make optimal choices under pressure.

Integrating Your Model

For developers interested in testing their own systems, the process is streamlined into three main steps. First, the model must be deployed behind an HTTP endpoint capable of receiving maze states and returning directional decisions. Second, users run the benchmark suite locally to ensure their model adheres to the timing and output specifications. Third, they submit their results through the project's contribution workflow.

Those who do not wish to host their own infrastructure are not left out. The project maintainers have invited users with models on public APIs, such as Opper, to open an issue. The team will then run the tests on their behalf, ensuring that the barrier to entry remains low for smaller research groups or individual developers.

The project is licensed under AGPL-3.0, encouraging collaborative improvement and derivative works. By opening both the code and the replay data, jevman fosters a community-focused approach to AI evaluation. It serves as a litmus test for decision models, highlighting strengths in pathfinding and predator avoidance without requiring massive computational resources.

As the leaderboard grows, it will likely reveal interesting contrasts between different model architectures. Whether a model prioritizes aggressive pellet collection or cautious retreat, the data will provide a clear picture of its strategic biases. For the AI community, jevman offers a playful yet rigorous arena to compare decision-making prowess against one of gaming's most enduring challenges.