DeepSeek Rises in Black-Box Security Benchmark

Overview

A rigorous new security benchmark focusing on black-box vulnerabilities has reshaped the understanding of cost-performance ratios in modern large language models. DeepSeek’s V4.1 Flash variant emerged as a standout performer, securing a score of 57 out of 100 problems. This achievement places it competitively among the leading models while highlighting a dramatic disparity in pricing efficiency compared to industry titans.

The benchmark, which tests models against real-world bug bounty findings without providing source code or answer keys, revealed that newer frontier models from major providers were unable to complete the evaluation. Their stringent safety controls blocked the offensive security tasks required by the test, effectively removing them from direct comparison in this specific domain.

Benchmark Methodology and Scope

The evaluation framework was constructed to simulate high-stakes penetration testing scenarios without compromising the integrity of the target systems. Researchers rebuilt 100 laboratory environments based on actual vulnerability disclosures. These environments were designed to be opaque to the AI models, meaning no source code or solution manuals were provided.

The target system independently verifies every proof submitted by the model.
The test environment ensures that models must rely on reasoning and exploitation logic rather than memorization.
All models are evaluated under identical constraints to ensure a fair comparison of their offensive security capabilities.

This approach creates a black-box assessment, mirroring real-world conditions where security professionals must find flaws without prior knowledge of the internal architecture. The decision to use real findings from bug bounty programs adds a layer of authenticity that synthetic datasets often lack. It measures how well models can generalize learned security principles to novel, albeit simulated, attack surfaces.

Performance Rankings and Key Findings

The results from the first 15 models tested show a tight competition at the top, with Opus 5 and Grok 4.7 tying for the highest score of 63 out of 100. Following closely, Grok 4.6 achieved a score of 62, and MiMo V2.6 Flash secured 60 solutions. DeepSeek’s entry, V4.1 Flash, solved 57 problems, demonstrating a robust capacity for complex security reasoning.

"The gap between the top performers and DeepSeek is narrow, but the disparity in cost is staggering."

While Opus and Grok lead in raw problem-solving ability, the cost-per-token metrics tell a different story about accessibility and efficiency. The benchmark highlights that high scores do not necessarily correlate with high prices, challenging the assumption that the most expensive models are always the most capable in specialized security tasks.

Opus 5 and Grok 4.7: 63/100 problems solved.
Grok 4.6: 62/100 problems solved.
MiMo V2.6 Flash: 60/100 problems solved.
DeepSeek V4.1 Flash: 57/100 problems solved.

The performance distribution suggests that mid-tier models are closing the gap with elite products. For organizations that do not require the absolute maximum score, these alternatives offer a viable path to reducing operational expenses without significant sacrifices in capability.

The Cost Efficiency Disparity

Perhaps the most striking data point from the benchmark is the extreme difference in cost between the top-tier models and their competitors. Grok 4.7, which tied for first place, achieved its score for less than a quarter of the cost of Opus. This indicates that the marginal gain in problem-solving ability from Opus comes at a premium that may not be justified for many use cases.

The contrast becomes even more pronounced when comparing MiMo V2.6 Flash to Opus. MiMo came within just three solutions of the top score. However, the financial implications are vast: MiMo cost $6.76 to run the benchmark, whereas Opus cost $569.08 for the same task.

This represents a cost reduction of over 98% for a negligible drop in performance.
Such efficiency gains are critical for enterprises processing large volumes of security queries.
The data supports a shift toward value-based selection rather than prestige-based selection.

For teams integrating AI into their security workflows, these figures suggest that DeepSeek and similar cost-effective models can handle a significant portion of routine black-box analysis. The savings generated could be redirected toward other critical infrastructure or expanded testing scopes.

Exclusion of Frontier Models Due to Safety Controls

An important observation from the study is the absence of the newest frontier models from the leading technology companies. These models did not receive low scores due to a lack of capability; rather, they were unable to complete the benchmark because their safety controls refused the nature of the tasks.

The benchmark requires the generation of exploit code and offensive security strategies. Many commercial models are fine-tuned to decline requests that could be interpreted as facilitating malicious activity, even in a controlled, defensive research context.

Safety filters prioritized responsible AI usage over capability demonstration.
This creates a blind spot in standard benchmarks that do not account for refusal rates.
The exclusion highlights a tension between open security research and corporate liability management.

This phenomenon raises questions about the readiness of these models for real-world red-teaming applications. If a model refuses to engage with hypothetical attacks, its utility in offensive security planning is limited, regardless of its theoretical intelligence. The benchmark thus serves as a filter for models that are not only intelligent but also willing to operate in the gray areas of security research.

Implications for the Security Industry

The findings from this benchmark have significant implications for how organizations select AI tools for security operations. The traditional hierarchy, which places expensive, heavily regulated models at the top, is being challenged by data showing superior efficiency from alternative providers.

DeepSeek’s strong showing reinforces its position as a serious competitor in the general-purpose AI market. Its ability to perform well in a specialized, reasoning-heavy domain like black-box security suggests that its underlying architecture is highly adaptable. This versatility could make it a preferred choice for startups and mid-sized firms looking to deploy AI-driven security solutions without incurring enterprise-level costs.

Furthermore, the exclusion of other major models due to safety guardrails indicates a need for more nuanced evaluation metrics. The industry may need to develop dual-track benchmarks that separately measure capability and compliance. This would allow buyers to see both the maximum potential of a model and the practical limitations imposed by its safety tuning.

Conclusion

The release of these benchmark results provides a clearer picture of the current AI landscape in the context of cybersecurity. While Opus and Grok remain leaders in raw performance, the cost advantages of models like DeepSeek V4.1 Flash and MiMo V2.6 Flash are undeniable. The near-100% cost savings for minimal performance loss offer a compelling argument for diversifying AI procurement strategies.

As the field evolves, the definition of a "top-tier" model will likely expand beyond simple accuracy metrics to include cost-efficiency and operational flexibility. The refusal of frontier models to participate in this type of offensive security work further underscores the importance of choosing tools that align with the specific demands of security research. For now, the data suggests that high performance and affordability are no longer mutually exclusive.