SoK: DARPA's AI Cyber Challenge (AIxCC) — Competition Design, Architectures, and Lessons Learned

Open ScienceSoK Artifact¶

Some materials respect DARPA's official release timeline and are not yet public.

Currently available

• Questionnaires, meeting notes, sanitized experiment data (PF / MR / CC), and analysis scripts — Zenodo bundle at zenodo.org/records/20367274.
• AIxCC Final Challenge Set, official release at archive.aicyberchallenge.com/challenges.

Pending release

• Raw competition data (submission logs, OTEL traces, scoring breakdowns).
• CRUMBS — the organizers' competition data analysis framework.

Open ScienceCollection of Teams' Public References¶

Open ScienceOSS-CRS & CRSBench¶

OSS-CRS

OpenSSF CRS orchestration platform which integrates the bug-finding and patch components from every AIxCC finalist CRS into a single open-source pipeline.

• Official website: oss-crs.openssf.org
• GitHub: ossf/oss-crs

CRSBench

The matching benchmark suite for OSS-CRS, bundling the AIxCC Final Challenge Set, the AIxCC Exhibition Round challenge set, and Team Atlanta's n-day-derived challenges.

• Official website: oss-crs.openssf.org/crsbench
• GitHub: sslab-gatech/CRSBench
• HuggingFace dataset: sslab-gatech/crsbench-dataset

Scoring Rules Explanation¶

The scoring system is centered around a developer-centric principle: reward outcomes that benefit project developers and penalize behaviors that would burden them. Organizers use this principle to determine when and to what extent should reward be given for each capability:

• A single PoV is not highly rewarded since it only demonstrates the vulnerability;
• Patches score highest as they directly solve security issues;
• SARIF assessments are useful but only offer non-core, semi-subjective report validation, therefore scoring lowest;
• Bundling saves developer investigation time when correct (rewarded), but largely wastes their efforts and emotions when incorrect (penalized).

Based on those weights, organizers introduce additional scoring mechanisms, like accuracy penalty and time-decay, to ensure competition fairness and balance research incentives with practicality.

Team Score Hierarchy

A team's total score is the sum of individual challenge scores:

Each challenge score equals the accuracy multiplier times the sum of four capability scores:

Accuracy Multiplier

The accuracy multiplier AM serves as a global button to encourage teams to balance effectiveness with reliability. Teams with 100% accurate submissions receive no penalty (AM = 1), while lower accuracy results in a reduced multiplier:

The quartic formula strongly penalizes low accuracy to discourage impractical strategies, while tolerating reasonable error rates to enable research innovation. For instance, at 90% accuracy AM ≈ 0.9999 (nearly no penalty), while at 50% AM = 0.9375 and at 40% AM = 0.8704 (significant score reduction).

Each submission is classified as accurate, inaccurate, or neutral, where neutral submissions do not affect accuracy calculations. For a given capability, the classification rationale is to reward correct, non-duplicate submissions, penalize incorrect ones, and not penalize partially correct or duplicate submissions that provide real-world value:

• PoV. One reproducible PoV per vulnerability is accurate; irreproducible PoVs are inaccurate; reproducible but duplicate PoVs are neutral, since collecting diverse PoVs can benefit patch validation in practice.
• Patch. First, patches that fail to apply, build, or remediate any PoV are inaccurate. Then, the patches that fail functionality tests are neutral. This considers that CRSs may not have access to full testing functionality in practice and a CI system can help catch such issues. Finally, among passing patches, a minimal covering set is selected as accurate while the rest are inaccurate.
• SARIF and Bundle. Only fully correct submissions are accurate; others are inaccurate.
• Server errors and schema mismatches. neutral.

Time-Decayed Scoring Design

Teams' submissions earn fewer points over time, with up to 50% reduction at the deadline. \(\mathit{Score}_{\text{PoV}}\), \(\mathit{Score}_{\text{Patch}}\), and \(\mathit{Score}_{\text{SARIF}}\) share a common time-decay formula:

where remaining time is from submission to deadline, and total duration is the challenge window. \(\tau\) ranges from 1.0 (immediate) to 0.5 (at deadline). The weight for PoV, Patch, and SARIF are 2, 6, and 1, respectively. For total duration, PoV and Patch submissions are measured from challenge start to deadline (full-mode for 12 hours while delta-mode for 6 hours), while SARIF is from broadcast time.

Note that for each vulnerability, only the last submission is scored, so earlier submissions can be superseded by later ones, allowing teams to revise their submissions with a certain penalty.

Bundle Scoring

A bundle reports pairings among PoV, Patch, and broadcast SARIF that a team identifies as addressing the same vulnerability. Unlike other capabilities, bundles can yield negative scores:

The sign is positive if all claimed pairings are correct, negative if any is incorrect. The more correct pairings, the higher the score (up to 7 points for vulnerabilities that have a SARIF broadcast, 4 for those without). CRS-generated SARIF may be included in bundles but does not affect scoring. Bundle scoring is indirectly affected by time-decay through the underlying PoV and Patch scores.

Patch Selection and Validation

Since a single patch may remediate multiple vulnerabilities, organizers must select which patches to credit. The selection algorithm identifies a minimal covering set: the smallest number of patches that collectively fix all validated vulnerabilities. When multiple patches cover the same vulnerability, specificity is preferred: patches fixing fewer vulnerabilities are chosen over broader ones, rewarding precise, targeted fixes. Patches not selected into this minimal set count as inaccurate.

To validate whether a patch truly remediates a vulnerability, organizers use all PoVs submitted by all teams, plus organizer-created ones, as test cases. A patch must remediate every PoV targeting its claimed vulnerability to be considered valid. This cross-team validation also reflects one collaboration approach among CRSs in practice.

Challenge Project (CP) Details¶

Detailed information about each Challenge Project (CP) in the AIxCC final round.

▲ delta-mode   □ full-mode. Cutoff: date of the latest upstream commit incorporated into the organizer's CP repository. Build time measured on AMD EPYC 7452 (128 cores), 512 GB RAM, Ubuntu 22.04.

Challenge Project Vulnerability (CPV) Details¶

Detailed information about each Challenge Project Vulnerability (CPV) in the AIxCC final round.

▲ delta-mode   □ full-mode. Multi-CWE CPVs span multiple rows.

SARIF Validation Techniques¶

The table below summarizes how each finalist team validates broadcast SARIF reports.

^* Heuristic-based follows developer-defined workflows; LLM-based relies on LLM queries.
^† Verifies SARIF report format and checks whether referenced artifacts (files, functions or lines) exist before main validation.
^‡ Additional code context dynamically requested by LLM.

SARIF Broadcast Details¶

Detailed information about each SARIF broadcast in the AIxCC final round.

▲ delta-mode   □ full-mode   ⊝ false-positive broadcast (no underlying CPV).

0-Day Details¶

Detailed information about 0-day vulnerabilities discovered during the competition.

▲ delta-mode   □ full-mode.

Token Consumption Details¶

The figures below show the per-model token consumption and the input-to-output token ratio per model. Token data is derived from OpenTelemetry (OTEL) logs collected by the organizers. As the collected data was incomplete due to the competition environment, the figures do not present full usage but serve as a lower-bound reference.

Figure C1. Token consumption (input + output) per model by team.

Figure C2. I/O token ratio per model by team.

Submission Timing¶

The figures below show per-team submission timing. Most teams front-load within the first quarter, consistent with time-decay scoring. SARIF is the fastest (avg 5.6%) as it triggers upon broadcast; bundles are the latest (avg 35.8%) as they depend on prior submissions and are scored at the end. Artiphishell is a notable outlier, spreading submissions across the entire window.

(a) PoV. n = 2519, avg = 25.0%

(b) Patch. n = 283, avg = 23.8%

(c) SARIF. n = 48, avg = 5.6%

(d) Bundle. n = 99, avg = 35.8%

Each task has a fixed time window (12h for full-mode, 6h for delta-mode); 0% is when the task opens, 100% is the deadline.

CWE-Wise Performance Analysis¶

The heatmaps below show CWE-wise team performance for PoV generation and patch generation, respectively.