PostHog's Jeeves Brings Reasoning to AI Decision Models

Overview

PostHog has released Jeeves, an open-source reasoning model designed to improve the accuracy of Jev-style decision classifiers. By integrating chain-of-thought reasoning directly into the decision-making pipeline, Jeeves addresses a persistent limitation in calibrated probability models: low raw accuracy on complex or out-of-domain tasks.

The project introduces a 9-billion-parameter variant based on Qwen3.5, fine-tuned with Low-Rank Adaptation (LoRA) and a specialized pointer head. Unlike traditional approaches that treat reasoning and decision-making as separate stages, Jeeves employs a diffusion drafter and a training methodology called CISPO to ensure the model "thinks" before it outputs a final prediction.

The Problem with Calibrated Decisions

Jev-like models are valued in enterprise AI pipelines because they provide calibrated decision probabilities. This means when a model predicts a 70% confidence level, it is correct approximately 70% of the time. However, these models often struggle with accuracy compared to pure reasoning models. Consequently, many production systems rely on heavier reasoning models as fallbacks, which increases latency and computational cost.

Jeeves aims to bridge this gap by training a Jev-compatible model to perform internal reasoning. This allows the system to maintain the probabilistic guarantees of Jev while achieving accuracy levels closer to standard large language models. The architecture supports multiple question types in a single request, including yes/no decisions, multiple-choice selections, and rating scores.

Performance and Benchmarks

In initial evaluations, Jeeves demonstrated significant improvements over existing baselines, particularly on tasks it had not encountered during training. On the overall test set, which includes out-of-domain and held-out items, Jeeves achieved an accuracy of 0.889, surpassing Kev-9B at 0.822 and the original Jev model at 0.857.

Performance on JevBench, a standardized benchmark for these types of classifiers, was equally impressive. Jeeves scored 0.935 on the public tiers, compared to 0.866 for Jev. The model showed notable strength in handling held-out rule structures and contrastive policies, where it achieved perfect or near-perfect scores of 1.000 and 1.000 respectively.

Benchmark	Kev-9B	Jev	Jeeves
Test Overall	0.822	0.857	0.889
JevBench Overall	0.715*	0.866	0.935
JevBench Hard	0.451*	0.730	0.865
Held-out Rule Structures	-	0.885	1.000

*Kev-9B JevBench results are approximations based on Kev-8B data, as official figures were not published.

Without the reasoning component, the same checkpoint scored 0.804 on the test split. Enabling thinking raised this to 0.840, confirming that the reasoning step contributes directly to performance gains.

Technical Architecture and Speed

The core innovation lies in the use of a block-4 diffusion drafter. This component works in tandem with the CISPO training method to guide the model's latent reasoning process. The model runs on CUDA, utilizing Hopper architecture features for FP8 kernel optimization, which helps manage the computational overhead of generating long reasoning chains.

Latency metrics indicate a trade-off between depth of thought and speed. On an NVIDIA H100 GPU:

• Requests without thinking process in approximately 0.3 seconds.
• Requests with full reasoning have a median latency of 3.3 seconds.

Developers can reduce latency by truncating the chain length, allowing for flexible deployment based on specific use-case requirements.

Implementation and Usage

PostHog has released the full training code, along with train, dev, and test datasets, under an open-source license. The project includes a Python SDK that serves as a drop-in replacement for Jev’s existing typesafe-sdk, facilitating easy migration for current users.

The model accepts requests in a JSON format that defines the current state and a dictionary of questions. Each question specifies its type (noul, choice, or score) and criteria. The response includes not only the final answer but also detailed probabilities and confidence scores for each option.

{
  "answers": {
    "department": {
      "type": "choice",
      "choice": "billing",
      "confidence": 0.19,
      "probabilities": { "returns": 0.4, "shipping": 0.14, "billing": 0.46 }
    },
    "escalate": { "type": "noul", "noul": 0.72 },
    "frustration": {
      "type": "score",
      "score": 1.5,
      "probabilities": { "0": 0.04, "1": 0.43, "2": 0.54 }
    }
  }
}

To get started, users require Python 3.12 and a CUDA-capable GPU. The model weights are available via Hugging Face, and the inference server can be launched with a single command specifying the model and drafter paths.

Jeeves represents a significant step toward making calibrated AI decisions both accurate and efficient. By proving that reasoning can be integrated into lightweight, probabilistic classifiers, PostHog offers a viable path for enterprises that need both speed and reliability in their AI pipelines.

The project acknowledges inspiration from the Kev initiative and continues to evolve through community contributions. With its open weights and comprehensive documentation, Jeeves invites researchers and engineers to experiment with reasoning-enhanced decision models in real-world applications.