TokenWave

SpecEngineeringBench

Evaluating Coding Agents as
Autonomous Software Test Engineers

Watch the overview

Code and data require repository access.

Xingjian Wang1, Shijian Wang3, Minghao Liu2, Ji Zou3, Zetao Yang1, Xinping Lei1, Yiming Liang1,
Shen Yan3, Enduo Zhao1, Xiaoyang Shen1, Weihao Xuan1, Jiaheng Liu3, Ge Zhang1

12 2077AI3 Independent Researcher

100benchmark instances
78Python repositories
531recorded defect occurrences
16application categories

SpecEngineeringBench in 92 seconds

THE BENCHMARK

Can an agent test a release
from its requirements?

SpecEngineeringBench evaluates the work that starts before a bug report: deciding which behaviors need testing, finding the relevant implementation, and turning failures into reproducible defect reports.

Agents receive a product requirements document (PRD), an unrepaired Python repository, and a configured Docker environment. Each instance contains 4–15 recorded defects. The agent chooses what to test and where to investigate.

Product requirementsRepository snapshotDocker runtime
01

Decide what needs testing

Translate required behaviors, boundary conditions, and error handling into concrete test assertions.

02

Find the relevant implementation

Investigate the repository without an issue description or designated files and functions to inspect.

03

Investigate multiple defects

Each snapshot contains several coexisting bugs. Finding an initial failure leaves further defects to discover and localize.

Figure 1. Issue-guided and target-guided benchmarks compared with requirements-driven autonomous testing in SpecEngineeringBench.
Figure 1. Conceptual comparison of representative benchmark settings and SpecEngineeringBench.

Two independent tasks

Test authoring and defect diagnosis are evaluated separately on the same benchmark instances.

01

Test generation

Submit an executable test suite for the required functionality. The evaluator replays the same tests on the defective and repaired snapshots to measure validity, defect exposure, and coverage.

Head Pass RateFail-to-Pass RateCoverage
02

Defect discovery & reporting

Explore the repository, write and run test probes, and report suspected defects with reproducible logs and source locations. Reports are scored at file, function, and edit-hunk granularities.

Localization recallLocalization precisionDistinct bugs

Leaderboard

Seven coding agents, evaluated on test generation and defect discovery.

Evaluation protocol ↗
91.47%Mean Head Pass Rate
80.48%Mean change-focused coverage
26.36%Mean Fail-to-Pass Rate

High test validity and coverage coexist with low defect exposure. Seven-agent means from Table 3.

Model results reported in the October 25, 2026 paper

Paper snapshot · 25 Oct 2026

DATASET

Benchmark statistics

Figure 2 · Repository inventory ↗

Application categories

Defects per instance

Number of instances

100 instances · Median 5 · Mean 5.31

80.4kmedian source lines
5median reference files
11.5median edit hunks

Per-instance medians · Table 2

All repositories

View repository inventory
RepositoryCategoryInstancesDefectsSource kLOC

Source kLOC is the median across each repository’s released defective snapshots. Defects are recorded occurrences, counted separately in each instance.

RESULTS & ANALYSIS

Key findings

FROM THE PAPER

Case studies

Appendices C & D ↗
python-visualization / folium

Requirements → Tests → Defect report

episode-0001
GPT-6-Astra · Codex5 reference bugs2 independent tasks

Paper, code, and data

The paper describes the benchmark construction, evaluation protocol, and experiments. Code and data are pending release.

Code and data require repository access.

SpecEngineeringBench
Paper abstract

Autonomous software testing requires coding agents to translate functional requirements into executable tests and use the results to discover and validate defects. However, existing benchmarks evaluate coding agents on explicitly specified test generation tasks, commonly providing an issue description, a designated test entry point, or files to inspect, leaving the ability to determine what to test and where to investigate insufficiently evaluated. Relying on such clues bypasses the challenge of mapping requirements to implementations and proactively uncovering unknown defects. We propose SpecEngineeringBench, comprising 100 instances from 78 open source Python repositories across 16 application categories, with multiple recorded defects in each instance. Two independent tasks evaluate complementary capabilities: generating test suites and discovering defects through execution analysis. The evaluation combines test validity, defect detection, and coverage with localization precision and recall. Across seven frontier coding agents, results reveal that test validity and coverage decouple from defect exposure: agents achieve a 91.47% Head Pass Rate and 80.48% Change Focused Coverage alongside a 26.36% Fail-to-Pass Rate, while localization recall drops from 78.11% at the file level to 51.29% at the function level. Autonomous test generation improves file, function, and hunk localization recall by 4.99, 2.77, and 2.69 percentage points over relying solely on existing repository test suites.

Citation

@misc{wang2026specengineeringbench,
  title={SpecEngineeringBench: Evaluating Coding Agents
         as Autonomous Software Test Engineers},
  author={Wang, Xingjian and Wang, Shijian and Liu, Minghao
          and Zou, Ji and Yang, Zetao and Lei, Xinping
          and Liang, Yiming and Yan, Shen and Zhao, Enduo
          and Shen, Xiaoyang and Xuan, Weihao and Liu, Jiaheng
          and Zhang, Ge},
  year={2026},
  note={Preprint}
}