Open-Source Benchmark Tests AI Agents on Financial Due Diligence

Overview

A new open-source benchmark released by SecondState offers a rigorous test of how well large language models can perform financial due diligence. The project, titled the Finance Agents Benchmark (FAB), challenges AI systems to navigate a synthetic company data room filled with complex documents and legal rubrics. By standardizing the evaluation process, researchers and developers now have a structured environment to measure the practical reasoning capabilities of top-tier AI agents in high-stakes financial scenarios.

The benchmark currently focuses on a single fictional entity, Meridian Industrial Supply LLC. This company serves as the central subject for a comprehensive dataset comprising 50 distinct tasks, 160 supporting documents, and 231 specific grading criteria. The goal is to determine whether autonomous agents can accurately interpret financial records, identify risks, and answer precise questions without human intervention. This move toward synthetic but realistic data rooms addresses a critical gap in AI evaluation, where traditional benchmarks often fail to capture the nuances of professional financial analysis.

Methodology and Structure

FAB is designed as a two-part system that separates the evaluation dataset from the execution engine. The first component is a curated dataset containing agent instructions, relevant corporate documents, and detailed rubrics for assessment. The second component is an execution harness that runs AI agents against these tasks and automatically grades their performance. This separation allows developers to test various model configurations and agent architectures using the same consistent standards.

To participate in the benchmark, users must set up a local environment requiring Python 3.11 or higher, along with uv for dependency management and Docker or Podman for containerization. After cloning the repository, participants configure their API credentials in a local environment file. The tooling supports both OpenAI and Fireworks AI models, enabling a wide range of commercially available large language models to be tested within the same framework. This accessibility encourages broader community participation and ensures that results are comparable across different provider ecosystems.

The evaluation process is stringent. Agents are given access to a sandboxed data room and must retrieve, interpret, and synthesize information to complete specific tasks. Each task is evaluated against multiple criteria, and a task is only considered passed if every single criterion is met. This all-or-nothing grading approach ensures that partial credit does not mask significant reasoning gaps, providing a more accurate reflection of an agent’s reliability in real-world due diligence situations.

Benchmark Results and Performance

The initial release of FAB includes results from four prominent models, each evaluated across three trials on all 50 tasks, generating a total of 600 answers. The performance of these models was judged by gpt-6-luna operating at maximum reasoning capability, serving as an authoritative evaluator for the synthetic legal and financial language used in the tasks. The data reveals a competitive landscape where top models perform similarly but exhibit distinct strengths and weaknesses in handling complex document retrieval and logical deduction.

Model	Task Pass Rate	Criterion Pass Rate	Passed at Least Once	Passed All Three
DeepSeek V4.1 Flash	60.0%	81.0%	38/50	23/50
GPT-6 Sol	58.7%	83.4%	35/50	24/50
GPT-6 Luna	50.7%	79.5%	33/50	19/50
GLM 5.3 Flash	47.3%	76.2%	30/50	18/50

DeepSeek V4.1 Flash leads in task-level pass rates, while GPT-6 Sol demonstrates the highest consistency in meeting individual grading criteria.

The results indicate that while many agents can successfully complete nearly 60% of the tasks, achieving consistent reliability across multiple trials remains a significant challenge. DeepSeek V4.1 Flash achieved the highest task pass rate at 60%, suggesting strong capabilities in navigating the synthetic data room. However, GPT-6 Sol showed a slight edge in criterion-level accuracy at 83.4%, highlighting a potential difference in how these models handle detailed verification requirements. Models like GLM 5.3 Flash, while capable, lagged slightly behind the leaders, underscoring the difficulty of the domain.

These findings are crucial for the development of enterprise-grade AI agents. Financial due diligence requires not just general knowledge but precise adherence to factual constraints and legal standards. The benchmark provides granular insights into where current models excel and where they are prone to errors, such as missing subtle contractual obligations or misinterpreting financial statements.

Resources and Availability

The FAB project is committed to transparency and reproducibility, making its code and data widely available to the research community. The codebase is licensed under the permissive MIT License, encouraging adoption and modification by developers and enterprises. In contrast, the data, tasks, rubrics, and results are released under CC BY 4.0, allowing for broad academic and commercial use while requiring appropriate attribution.

Researchers and practitioners can access detailed documentation through a comprehensive walkthrough that guides users through setup, dataset inspection, and the grading process. Additionally, full results are published in a detailed report, providing criterion verdicts, judge reasoning, and usage statistics. The dataset is also hosted on Hugging Face, offering versioned data rooms and loadable agent traces for deeper analysis.

Citing this work requires including specific references to the FAB project and noting any code or dataset revisions used during experimentation. By providing these resources, SecondState aims to establish a standardized metric for evaluating AI agents in financial contexts, driving progress toward more reliable and trustworthy autonomous systems in the fintech sector.