New CWE-bench v2 Tests AI Models on 100 Cyber Security Tasks

Collinear AI has launched CWE-bench v2, a public benchmark created to test the limits of advanced artificial intelligence models in cyber security. The benchmark provides a set of 100 tasks that allow developers to evaluate how well their models perform.

How the Testing Works

During the evaluation, an AI agent receives a copy of a real open-source software repository along with a short instruction. The main goal for the agent is to find and fix a security weakness without breaking the existing code functionality. The scoring system is fully programmatic, ensuring that the results are deterministic and reproducible without extra costs. A held-out set of tasks is also maintained to track progress on challenges that models have not seen before.

Leaderboard and Performance Results

The 100 tasks cover 91 different security weakness classes. Out of 11 evaluated models, Gemini 4 Argon leads the leaderboard with a 67.8% pass rate on the first attempt, closely followed by Claude Opus 5.5. At the same time, even the top-performing model fails nearly a fifth of the set on every attempt, showing that significant challenges remain. On the cost front, models like DeepSeek V4.1 Flash offer lower-cost options, while higher-end models achieve better pass rates at increased expenses.

Open Access and Environment

Every task is built as a self-contained environment using container technology, allowing users to run the entire benchmark directly from its repository. The tasks are also tagged for reinforcement learning, meaning the same environments and verification tools can be used to train defensive security agents in the future.