Can coding agents become dependable digital engineers?
SWE-CoWork evaluates coding agents as digital employees in software organizations. From a brief handoff, they must recover requirements, coordinate with colleagues, and carry the work through implementation, verification, and reporting.
People set direction. Agents carry the work through.
Our goal is to let people focus on priorities, major decisions and final review, while agents manage the investigation, coordination and follow-through of everyday engineering.
ONE REQUIREMENT / MEM-227Illustrative reconstruction
A Mempool task: decide what the audit summary should return when no saved audit exists.
START WITH THE HANDOFF
The handoff identifies the project and where to look next. The agent uses the board, project records and colleagues to work out which requirements remain open.
Model results in the CoWork setting. Click a metric heading to sort.
#
Model
01
Claude-Opus-5.5
63.84%
93.24%
6.93%
02
GPT-6 Astra
62.91%
89.15%
7.92%
03
Seed Evolving
39.09%
94.48%
0.99%
04
Qwen3.8-Max
35.70%
91.80%
0.00%
05
GLM 5.3
31.27%
87.72%
0.99%
06
Qwen3.8-Flash
30.70%
93.40%
0.00%
07
DeepSeek V4.1 Flash
28.20%
94.30%
0.00%
08
Kimi K3
27.08%
84.88%
0.99%
09
GLM 5.2
24.40%
95.20%
0.00%
10
Seed 2.1 Pro
21.01%
81.85%
0.00%
11
DeepSeek V4 Pro
13.04%
90.21%
0.00%
F2P measures the fraction of tests for new behavior that pass.
P2P measures the fraction of existing-behavior tests that continue to pass.
Full-task pass requires all F2P and P2P checks to pass in the same run.
F2P and P2P are macro-averages of per-task test pass rates. Full-task pass is a percentage of tasks. Source: the reported 101-task release, October 2026. Experimental settings ↗
02 / INSIDE ONE TASK
Follow one requirement. See what changes at each step.
Follow one Mempool requirement from an owner’s clarification to a patch and local checks.
The same change can start in two different ways. An issue-style task supplies the specification up front. In SWE-CoWork, the agent must establish it from people and records, then carry it through the code and checks.
MEM-227 / TASK 078 Condensed from the supplied replay
01 / 05 · Read the handoff
01 / READ THE HANDOFF
A starting point. Not a full specification.
The agent inherits open work. The handoff identifies the project and available sources, but the agent still has to establish the requirements.
NEXT QUESTIONWhat remains to be done, and where is the authoritative context?
An open follow-up. The compatibility contract needs confirmation.
YJ
Yuan JingResponsible colleague
trace the requirement
Ask about the confirmed output contract.
Records provide leads. The owner supplies the clarification.
03 / CONFIRM THE CONTRACT
The answer determines what must be built.
The owner confirms that the summary must use persisted audit data. If none exists, it must return null. It must not synthesize a summary from transactions.
CONFIRMED REQUIREMENTStored audit data or null. No synthesized fallback.
Which data source backs the audit summary? What should it return when no audit is stored?
YJ YUAN JING / OWNER
Only persisted audit data. If no audit exists, return null. Do not synthesize a summary.
Audit exists → stored auditNo audit → null
Conversation condensed for clarity from the recorded exchange.
04 / CHANGE THE CODE
Carry the clarification into the implementation.
The patch removes a synthesized-summary fallback and uses the persisted audit result. The diagram shows the behavior change; the source view contains the recorded diff.
IMPLEMENTATION TARGETMake the code match the owner’s confirmed contract.
Build a summary from transactions when no audit exists.
CONFIRMED BEHAVIOR
Stored audit→return audit
No audit→return null
A schematic of the behavior, not a verbatim code listing.
05 / CHECK THE CONFIRMED BEHAVIOR
Test both cases. Check the results.
The agent writes two tests for MEM-227. One checks that an existing audit is returned; the other checks that a missing audit produces null without generating a replacement summary. Both tests pass.
WHAT THE TESTS CHECKDoes the code handle both cases as the colleague specified?
The two checks correspond to the two cases confirmed by the colleague.
03 / WHAT THE EVALUATION REVEALS
Finding the requirements is only part of the job.
01
Requirement acquisition tracks delivery.
Across eleven models, the share of colleague-held requirements obtained is strongly associated with F2P performance. The number of tool calls has little association.
0.94Requirements obtained
0.08Tool calls
Spearman correlation with F2P across models. This association does not establish causation.
Figure 5 · requirement acquisition and tool use
02
Received does not mean implemented.
In the Claude-Opus-5.5 analysis, 56% of 976 failed F2P nodes with linked requirements were assigned to implementation, after the requirement reached the agent.
42% Ask56% Implement2% Discovery · 0% Read
Requirement-linked trajectory analysis, not the distribution of all task outcomes.
03
A full specification does not finish the task.
With complete requirements supplied at the start, GPT-6 Astra’s F2P score rises from 62.91% to 78.84%. Its full-task pass rate stays at 7.92%.
CoWork62.91%
→
Oracle-Spec78.84%
F2P
Same model, repositories, tests, scaffold and budget. The input specification changes; this is not a training result.
04 / THE VISION
Toward digital engineers people can delegate to.
Software work spans people, records and code. Reliable delegation means carrying an assignment from a brief handoff to a verified result.
Our goal is for people to set direction, choose priorities and review outcomes, while agents recover context, coordinate dependencies and deliver the work. Four capabilities are central to that goal.
01 / CONTINUITY
Keep context current.
Reconcile boards, versioned documents, conversations and code. Check which decisions still apply, and revise the plan when new evidence arrives.
02 / INITIATIVE
Find and clarify the work.
Start from a brief handoff, identify open requirements and find the responsible colleagues. Ask with code or execution evidence, while keeping unrelated work out of scope.
03 / CONCURRENT PROGRESS
Keep work moving.
When an answer is deferred or a question is referred to someone else, advance other actionable work. Return to the pending thread and resolve the dependency within the available budget.
04 / PERSISTENCE
Track every requirement to delivery.
Keep each requirement in view through clarification, implementation and verification. Finish with a handoff that accurately records delivered behavior, test results and remaining issues.
SWE-CoWork measures progress toward this goal by linking distributed requirements to executable checks and tracing what reaches the final delivery.
THE BENCHMARK
Real code. An organization to navigate.
Each assignment pairs a real upstream repository with an authored workplace history. Colleagues have defined responsibilities, bounded local knowledge and disclosure rules.
The agent starts with a brief handoff, reconciles records, asks colleagues, changes the code and checks its work. New evidence can send it back to an earlier step.
Full figure captions are retained from the original project page. Select any figure to inspect it at full resolution.
Figure 1 The same Click requirement presented directly as an issue (left) and within an ongoing SWE-CoWork assignment (right). Knowledge is distributed among digital employees and workplace records; the agent recovers the requirement, consults the responsible colleague, implements and verifies the change, and updates its interpretation when new evidence appears.Figure 2 Composition of the 101 SWE-CoWork tasks: (a) primary language, (b) application domain, (c) production lines added, on a log axis, (d) verifier test nodes per task. Dashed lines mark medians.Figure 3 An owner clarification changes a timesheet export implementation in a recorded run (task 094, Kimai). The owner specifies that the user’s preference must be evaluated on every render, and the agent updates its implementation and tests.Figure 4 Five workplace information challenges and the judgments they require, organized by the timing, provenance, and authority of evidence. The panels use schematic situations and condensed workplace-record examples.Figure 5 Requirement acquisition, tool calls, and delivery across eleven models. (a) F2P against the share of colleague-held requirements disclosed during the run (Spearman ρ = 0.94). (b) F2P against median tool calls per run, on a log axis (ρ = 0.08).Figure 6 Model calls, token usage, and delivery. Each point pairs a model’s per-run usage median with its full-release F2P. The output axis uses a log scale. Dots and leaders mark exact values where nearby logos are offset. Seed Evolving input is approximate; its output median is unavailable.Figure 7 Tool use, context compaction, and delivery across nine models. Tool calls use a logarithmic axis; compactions use a linear axis to include zero. Dots and leaders mark exact values where nearby logos are offset.Figure 8 Workplace exploration, first code edit, and delivery. Full-release F2P against per-run medians of (a) workplace queries before the first code edit, (b) the tool-call index of that edit, (c) that index as a percentage of all tool calls in the run, (d) workplace queries over the full run, and (e) messages sent to colleagues. Panels (a–d) cover eight models; (e) covers the seven with a reported message median. Panel (c) uses the median of per-run percentages. All axes are linear. Dots and leaders mark exact values where nearby logos are offset.Figure 9 Requirement acquisition and loss in Claude-Opus-5.5. (a) Colleague-held requirements by acquisition state. (b) F2P pass rates by whether the requirement reached the agent. (c) Earliest observed loss stage for failed nodes.Figure 10 Follow-through and reporting in Claude-Opus-5.5. (a) Delivery claims compared with test outcomes. (b) Deferred threads and their follow-up. (c) Timing of colleague messages within a run.Figure 11 Claude-Opus-5.5 behavior profile. (a) Share of assessable runs by observed behavior, from ineffective to adaptive. (b) Spearman associations with per-task F2P, with exploratory 95% bootstrap intervals.Figure 12 Effort, outcomes and stopping in Claude-Opus-5.5. (a–c) Tool calls, colleague messages and hours against per-task F2P. (d) Distribution of F2P over the 101 tasks. (e) How runs ended, including stops with known, actionable gaps.Figure 13 Examples of evidence assessment in workplace records. (a) A published correction supersedes an earlier return shape. (b) Run identifiers expose a discrepancy in copied terminal output. (c) Output semantics and an open question about stream ownership.
@misc{lei2026swecoworkbenchmarkingcodingagents,
title={SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations},
author={Xinping Lei and Minghao Liu and Ruijie Zhu and Xiaohui Hu and Xingqiao Li and Ao Zhang and Wenhao Chai and Han Li and Tao Wang and Jingyuan Zhang and Enduo Zhao and Yunfeng Shi and Yiming Liang and Zetao Yang and Xingjian Wang and Yifan Yao and Xinkai Ma and Yingru Li and Tianhan Yang and Huan Zhou and Weiran Shi and Shen Yan and Jiaheng Liu and Stephen Huang and Eli Cheung and Weihao Xuan},
year={2026},
eprint={2611.20218},
archivePrefix={arXiv},
primaryClass={cs.SE},
url={https://arxiv.org/abs/2611.20218},
}