Decide what needs testing
Translate required behaviors, boundary conditions, and error handling into concrete test assertions.

Code and data require repository access.
THE BENCHMARK
SpecEngineeringBench evaluates the work that starts before a bug report: deciding which behaviors need testing, finding the relevant implementation, and turning failures into reproducible defect reports.
Agents receive a product requirements document (PRD), an unrepaired Python repository, and a configured Docker environment. Each instance contains 4–15 recorded defects. The agent chooses what to test and where to investigate.
Translate required behaviors, boundary conditions, and error handling into concrete test assertions.
Investigate the repository without an issue description or designated files and functions to inspect.
Each snapshot contains several coexisting bugs. Finding an initial failure leaves further defects to discover and localize.
Test authoring and defect diagnosis are evaluated separately on the same benchmark instances.
Submit an executable test suite for the required functionality. The evaluator replays the same tests on the defective and repaired snapshots to measure validity, defect exposure, and coverage.
Explore the repository, write and run test probes, and report suspected defects with reproducible logs and source locations. Reports are scored at file, function, and edit-hunk granularities.
Seven coding agents, evaluated on test generation and defect discovery.
High test validity and coverage coexist with low defect exposure. Seven-agent means from Table 3.
DATASET
Number of instances
100 instances · Median 5 · Mean 5.31
Per-instance medians · Table 2
| Repository | Category | Instances | Defects | Source kLOC |
|---|
Source kLOC is the median across each repository’s released defective snapshots. Defects are recorded occurrences, counted separately in each instance.
RESULTS & ANALYSIS
FROM THE PAPER
The paper describes the benchmark construction, evaluation protocol, and experiments. Code and data are pending release.
Code and data require repository access.

Autonomous software testing requires coding agents to translate functional requirements into executable tests and use the results to discover and validate defects. However, existing benchmarks evaluate coding agents on explicitly specified test generation tasks, commonly providing an issue description, a designated test entry point, or files to inspect, leaving the ability to determine what to test and where to investigate insufficiently evaluated. Relying on such clues bypasses the challenge of mapping requirements to implementations and proactively uncovering unknown defects. We propose SpecEngineeringBench, comprising 100 instances from 78 open source Python repositories across 16 application categories, with multiple recorded defects in each instance. Two independent tasks evaluate complementary capabilities: generating test suites and discovering defects through execution analysis. The evaluation combines test validity, defect detection, and coverage with localization precision and recall. Across seven frontier coding agents, results reveal that test validity and coverage decouple from defect exposure: agents achieve a 91.47% Head Pass Rate and 80.48% Change Focused Coverage alongside a 26.36% Fail-to-Pass Rate, while localization recall drops from 78.11% at the file level to 51.29% at the function level. Autonomous test generation improves file, function, and hunk localization recall by 4.99, 2.77, and 2.69 percentage points over relying solely on existing repository test suites.
@misc{wang2026specengineeringbench,
title={SpecEngineeringBench: Evaluating Coding Agents
as Autonomous Software Test Engineers},
author={Wang, Xingjian and Wang, Shijian and Liu, Minghao
and Zou, Ji and Yang, Zetao and Lei, Xinping
and Liang, Yiming and Yan, Shen and Zhao, Enduo
and Shen, Xiaoyang and Xuan, Weihao and Liu, Jiaheng
and Zhang, Ge},
year={2026},
note={Preprint}
}