Jeff Brings Fast Zero-Shot AI Decision-Making to Local Devices

Overview

A new open-source project called Jeff is bringing fast, zero-shot AI decision-making directly to local devices, challenging the assumption that heavy cloud infrastructure is required for real-time classification tasks. By fine-tuning smaller language models like Qwen3.5 and Gemma 4, Jeff delivers calibrated probability scores for specific options without generating verbose text responses.

The system operates with remarkable speed, processing decisions in approximately 22 milliseconds on high-end NVIDIA hardware and 28 milliseconds on Apple Silicon. This efficiency allows developers to embed sophisticated judgment capabilities into applications ranging from customer support routing to game AI, all while maintaining strict data privacy by running entirely on local hardware.

Technical Architecture and Speed

Jeff distinguishes itself by functioning as a direct classification engine rather than a generative text model. Users describe a situation and list available options in plain language; the model then returns a calibrated probability for each option in a single forward pass. This approach eliminates the latency associated with token generation and subsequent parsing, making it suitable for time-sensitive applications.

"These are very small models. They make extremely fast, well-calibrated judgement calls between options, and they slot easily into your local code."

The project supports three distinct question types to handle various decision-making needs:

• Choice: Selects one option from a list of up to 255 possibilities.
• Noul: Provides a yes/no answer, returned as a probability score.
• Score: Evaluates a position on a user-defined scale.

Several independent questions can be bundled into a single request, allowing the model to answer them simultaneously. This architecture ensures that the system remains lightweight while offering flexible integration through a request format compatible with existing tools like Jev.

Training Methodology and Data Integrity

One of Jeff’s most compelling features is its commitment to fully local training pipelines. The development process utilizes synthetic training data generated by open models, specifically Qwen3.8-Flash-Next, running on DGX Spark hardware. This eliminates reliance on closed-source model outputs for training data, addressing growing concerns about data provenance and intellectual property in AI development.

The training workflow is designed for efficiency on consumer and prosumer hardware. Fine-tuning the 0.8B parameter model takes approximately two hours on a single RTX PRO 6000 workstation GPU, while the larger 2B model requires about 3.5 hours. Testing is conducted on MacBook hardware, demonstrating that the final products are optimized for deployment on portable devices rather than exclusive server farms.

For scenarios where zero-shot accuracy is insufficient, the project supports rapid fine-tuning. In one documented case, a voice-navigation fine-tune improved held-out accuracy from 31.7% to 95.8% in under half an hour using a single GPU. This flexibility allows organizations to adapt the base models to highly specialized domains without extensive computational resources.

Performance Benchmarks and Real-World Tests

Benchmark results indicate that Jeff significantly outperforms its untrained counterparts while remaining competitive with much larger systems on specific tasks. Across five public benchmarks, the Jeff-Qwen3.5-2B model achieved an overall score of 83.1, surpassing the untrained Qwen3.5-2B baseline of 46.5 and approaching the published Jev score of 83.0.

On specialized datasets such as Financial PhraseBank and RAGTruth, the fine-tuned models demonstrated exceptional precision, scoring 96.3 and 88.9 respectively. However, the benchmarks also highlight the limitations of smaller models. On reasoning-heavy tasks like BBH and JudgeBench, Jeff trails behind larger models like AutoJev-27B, a gap expected given the disparity in model size and parameter counts.

To test zero-shot capabilities in dynamic environments, developers evaluated Jeff on classic video games including Doom, Frogger, and Pac-Man. The model was tasked with choosing legal moves based on textual descriptions of game states. While the random baseline performed poorly, Jeff’s zero-shot gameplay significantly outperformed random selection, though it did not match hand-coded rule bots. These tests illustrate the model’s potential as a "System 1" rapid-decision engine for interactive applications.

Deployment and Accessibility

Jeff is distributed as an independent project and is not affiliated with TypeSafe, the makers of Jev, though it adopts compatible API formats for ease of use. Models are available on Hugging Face in three variants: Jeff-Qwen3.5-0.8B, Jeff-Qwen3.5-2B, and Jeff-Gemma4-E2B.

Deployment is streamlined through Python tooling, with support for both NVIDIA GPUs via PyTorch and Apple Silicon via MLX. The quick-start process involves downloading checkpoints and launching a local server with minimal configuration. This accessibility lowers the barrier to entry for developers seeking to integrate AI-driven decision logic into local applications without managing complex cloud dependencies.

The project represents a shift toward pragmatic, efficient AI systems that prioritize speed and calibration over raw generative power. By proving that small, fine-tuned models can handle complex classification tasks locally, Jeff offers a viable path for privacy-conscious and latency-sensitive AI deployments.