Research note · October 2026

Own the work.
Don’t just patch the issue.

SWE-CoWork places a coding agent inside an organization of digital employees. Requirements sit with accountable colleagues and project records, and each one is tied to executable tests, so we can follow it from the first question to the final handoff.

SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations
26 digital employees per task·fixed executable contract
101real repositories
11programming languages
12application domains
THE QUESTIONCan an agent turn a brief handoff into verified delivery?
01 · The shift

The hard part starts
before the first line of code.

Most coding benchmarks hand an agent a packaged issue. SWE-CoWork hands it something closer to a Monday morning: a short onboarding note, a living repository, and a workplace where the missing context belongs to other people.

Issue-style SWE the usual benchmark
pallets / clickOpen issue

Output capture

Extend the result: preserve write order; expose stdout and stderr separately.

  • Specwritten up front
  • Peoplenone to consult
  • Contextall in the ticket
issue + repository→patch
SWE-CoWork work already in progress
Handoff“Finish the compatibility work.”Six minutes of context. The previous owner is unavailable.
BoardCLK-201 · Output capture
DocsDOC-11 · compare revisions
HistoryTrace the caller and prior behavior
Illustrative workflow
organization + evidence→verified delivery
“

The agent receives a brief handoff. It establishes each requirement with the colleague or record that holds it, and then carries that requirement through code, verification, and the final handoff.

02 · The loop

Six moves turn a handoff
into a delivery.

There is no prescribed order. New evidence can change the agent’s picture of the work, so a good run keeps circling between people, records, code, and checks.

01illustrative task loop
01 / 06
orient
board
docs
colleague
tests
START WITH A MAP

Read the handoff. Locate the work.

The agent identifies the target project, scope, and first leads before it spends effort on a patch.

onboarding noterepo statescope
Select any stage to explore
03 · The environment

Not a toy task.
A whole organization.

Every assignment is grounded in a real upstream repository and surrounded by an authored project history. Each digital employee has a defined responsibility, bounded local knowledge, and rules for what it can disclose.

2,626employee profiles across the benchmark26 digital employees × 101 tasks
40,836chat messagesplus 10,329 board cards
2,955median production linesacross 60 files, tests excluded
32median verifier nodes2,699 F2P · 562 P2P total
LANGUAGEStasks per language
JavaScript
20
Python
17
TypeScript
12
PHP
10
C#
9
Ruby
9
+ Go · Java · C++ · Rust · Kotlin 11 total
DOMAINSupstream repositories
Web & frontend26
Dev tools & compilers22
Data & databases11
Games & graphics10
Desktop apps7
Cloud & infra5
Finance & payments5
ML & AI systems4
+ security · mobile · observability · scientific 12 total
04 · The evidence

Partial competence is not
ownership.

Eleven models work against the same 101-task release contract. Switch the metric to compare new behavior delivered (F2P), existing behavior preserved (P2P), and assignments where every test passes in the same run.

CO-WORK OUTCOMEShigher is better · %
Claude-Opus-5.5
63.84
GPT-6 Astra
62.91
Seed Evolving
39.09
Qwen3.8-Max
35.70
GLM 5.3
31.27
Qwen3.8-Flash
30.70
DeepSeek V4.1 Flash
28.20
Kimi K3
27.08
GLM 5.2
24.40
Seed 2.1 Pro
21.01
DeepSeek V4 Pro
13.04
0255075100
REQUIREMENT ACQUISITION

Models that obtain more requirements deliver more.
More tool calls do not.

F2P against requirement acquisition or tool calls for eleven models
Show the data table
ModelMessagesTool callsHoursColleague req. obtainedAll req. obtainedF2P

Medians per run. Colleague: requirements held only by colleagues that were disclosed. All: adds public requirements, counted as obtained.

USAGE AND DELIVERY

Token use and model calls.
How do the medians compare with F2P?

View the figure ↓

F2P does not rise with the amount of interaction. The weakest models, such as Kimi K3 and GLM 5.2, make about 300 tool calls and stop well within the budget, at 24–27% F2P. The middle group, from Seed Evolving to DeepSeek V4.1 Flash, makes 500 to 1,600 tool calls, compacts its context up to 33 times, and obtains up to 59% of the requirements held by colleagues, yet reaches only 28–39%. These agents can already find the relevant colleagues and obtain much of the context, but struggle to keep track of each requirement from clarification through implementation and verification to the delivery report. The two leading models make fewer than 200 tool calls and reach 63–64%, and they share the same limitation on a smaller scale: in Claude-Opus-5.5 runs, 56% of failed nodes with linked requirements fail at implementation, and 31% of requirements reported as done fail their tests.

Claude-Opus-5.5 still uses about twice GPT-6 Astra’s input because of its context window. GPT-6 Astra’s window is about 258K tokens and its median peak context is 245K, so most runs are compacted; Claude-Opus-5.5 has a 1M-token window, a median peak of 483K, and never compacts.

Loading the usage medians…

Usage summaries cover nine of the eleven evaluated models. Token medians are unavailable for Seed 2.1 Pro and DeepSeek V4 Pro. F2P uses the full 101-task release. Dots and short lines mark exact values where nearby logos are offset. M = 1,000,000 tokens.

Show all usage medians
Per-run usage medians and full-release F2P
ModelModel callsInput (M)Output (M)Tool callsCompactionsF2P

≈ marks approximate input usage. — means unavailable.

WORKPLACE EXPLORATION

Workplace queries, first edits,
and messages to colleagues.

Enlarge the figure ↓

How much a model explores does not predict delivery. Claude-Opus-5.5 makes a median of only 56 workplace queries per run, while DeepSeek V4.1 Flash makes 572. The two leading models reach their first code edit at tool calls 67 and 82, still about a third of the way into their short runs. Kimi K3 and GLM 5.2 query and edit on a similar schedule but reach less than half the F2P. The clearest difference is colleague contact: they send medians of 8 and 5 messages, against 19.5 for Claude-Opus-5.5 and 18.5 for GPT-6 Astra. Asking colleagues is not sufficient on its own: GLM 5.3 sends the most messages, 21, and reaches 31.27% F2P.

Five panels comparing workplace queries, first-edit tool-call index and share, and messages to colleagues against F2P for eight models
Show all workplace exploration medians
Per-run medians and full-release F2P for eight models
ModelQueries before editingFirst-edit callFirst-edit shareQueries over the runMessages to colleaguesF2P
Claude-Opus-5.5398238%5619.563.84%
GPT-6 Astra526731%10118.562.91%
Seed Evolving39.58512.53%134—39.09%
Qwen3.8-Max7716323%170635.70%
GLM 5.37422713%2152131.27%
DeepSeek V4.1 Flash27270353%5724.528.20%
Kimi K337.510537%73827.08%
GLM 5.240.513034%100524.40%

First-edit share is the median of each run’s first-edit tool-call index divided by its total tool calls, expressed as a percentage. Query and message counts are medians per run. — means unavailable.

THE ORACLE-SPEC TEST

Hand over every requirement.
F2P rises about 16 points. Pass rates stay where they were.

Same model, repositories, tests, scaffold, and budget. The only change is that the complete requirements arrive at the start. With every requirement in hand, Seed Evolving still leaves 44% of F2P nodes unsatisfied and GPT-6 Astra 21%.

Seed Evolving
F2P
39.0955.76
+16.67 pp
P2P
94.4889.50
−4.98 pp
Pass
0.990.99
±0.00 pp
GPT-6 Astra
F2P
62.9178.84
+15.93 pp
P2P
89.1591.81
+2.66 pp
Pass
7.927.92
±0.00 pp
CoWorkOracle-Spec
05 · The diagnosis

Where does delivery
actually break?

Claude-Opus-5.5 has the highest F2P score, so its recorded trajectories become a microscope. The traces connect each failed test to its requirement, the colleague who holds it, the question, the patch, and the final handoff note. The agent finds the right people. It loses track of individual requirements afterward.

93%of first questions go to the accountable owneronly 3% of requirements held by colleagues sit with owners never contacted
153/450deferred threads are never revisitedmedian of 7.1 hours of budget left
31%of requirements reported as done fail their tests255 of 826 completion claims
REQUIREMENT ACQUISITION2,176 requirements held by colleagues
64%16%17%
Disclosed 1,400 Asked, no disclosure 357 Card not raised 363 Never contacted 56

Most undisclosed requirements belong to owners the agent already contacted: the relevant card was never raised, or the conversation ended without an answer.

WHERE FAILED NODES WERE LOST976 failed F2P nodes
Discovery
2%
Ask
42%
Read
0%
Implement
56%

42% of failed tests trace back to a requirement that stayed with a colleague. 56% trace back to a requirement the agent received and still did not satisfy.

THE TAKEAWAY

Asking is not acquiring.
Acquiring is not verifying.
Shipping is not finishing.

Dependable delivery means tracking every requirement until it is closed: raise it with its owner, revisit deferred questions, implement it, verify it against the tests, and report its true status in the handoff.

06 · The paper

Every figure,
straight from the paper.

All fourteen figures from the paper, rendered from the source files. Select a figure to enlarge it, or read the full PDF.

Figure 1 The same Click requirement presented directly as an issue (left) and within an ongoing SWE-CoWork assignment (right). Knowledge is distributed among digital employees and workplace records; the agent recovers the requirement, consults the responsible colleague, implements and verifies the change, and updates its interpretation when new evidence appears.
Figure 2 Composition of the 101 SWE-CoWork tasks: (a) primary language, (b) application domain, (c) production lines added, on a log axis, (d) verifier test nodes per task. Dashed lines mark medians.
Figure 3 An owner clarification changes a timesheet export implementation in a recorded run (task 094, Kimai). The owner specifies that the user’s preference must be evaluated on every render, and the agent updates its implementation and tests.
Figure 4 Five workplace information challenges and the judgments they require, organized by the timing, provenance, and authority of evidence. The panels use schematic situations and condensed workplace-record examples.
Figure 5 Requirement acquisition, tool calls, and delivery across eleven models. (a) F2P against the share of colleague-held requirements disclosed during the run (Spearman ρ = 0.94). (b) F2P against median tool calls per run, on a log axis (ρ = 0.08).
Figure 6 Model calls, token usage, and delivery. Each point pairs a model’s per-run usage median with its full-release F2P. The output axis uses a log scale. Dots and leaders mark exact values where nearby logos are offset. Seed Evolving input is approximate; its output median is unavailable.
Figure 7 Tool use, context compaction, and delivery across nine models. Tool calls use a logarithmic axis; compactions use a linear axis to include zero. Dots and leaders mark exact values where nearby logos are offset.
Figure 8 Workplace exploration, first code edit, and delivery. Full-release F2P against per-run medians of (a) workplace queries before the first code edit, (b) the tool-call index of that edit, (c) that index as a percentage of all tool calls in the run, (d) workplace queries over the full run, and (e) messages sent to colleagues. Panels (a–d) cover eight models; (e) covers the seven with a reported message median. Panel (c) uses the median of per-run percentages. All axes are linear. Dots and leaders mark exact values where nearby logos are offset.
Figure 9 Requirement acquisition and loss in Claude-Opus-5.5. (a) Colleague-held requirements by acquisition state. (b) F2P pass rates by whether the requirement reached the agent. (c) Earliest observed loss stage for failed nodes.
Figure 10 Follow-through and reporting in Claude-Opus-5.5. (a) Delivery claims compared with test outcomes. (b) Deferred threads and their follow-up. (c) Timing of colleague messages within a run.
Figure 11 Claude-Opus-5.5 behavior profile. (a) Share of assessable runs by observed behavior, from ineffective to adaptive. (b) Spearman associations with per-task F2P, with exploratory 95% bootstrap intervals.
Figure 12 Effort, outcomes and stopping in Claude-Opus-5.5. (a–c) Tool calls, colleague messages and hours against per-task F2P. (d) Distribution of F2P over the 101 tasks. (e) How runs ended, including stops with known, actionable gaps.
Figure 13 Examples of evidence assessment in workplace records. (a) A published correction supersedes an earlier return shape. (b) Run identifiers expose a discrepancy in copied terminal output. (c) Output semantics and an open question about stream ownership.
Figure 1

07 · The team

Authors

LEADING AUTHORS

Xinping Lei

CORE CONTRIBUTORS

Minghao Liu, Ruijie Zhu, Xiaohui Hu

CORRESPONDING AUTHORS

Ge Zhang, Jiaheng Liu, Weihao Xuan

CONTRIBUTORS

Xingqiao Li, Ao Zhang, Wenhao Chai, Han Li, Jingyuan Zhang, Enduo Zhao, Yunfeng Shi, Yiming Liang, Zetao Yang, Xingjian Wang, Yifan Yao, Xinkai Ma, Tianhan Yang, Huan Zhou, Weiran Shi, Shen Yan

CITATION
@misc{swecowork2026,
  title  = {SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations},
  author = {Lei, Xinping and Liu, Minghao and Zhu, Ruijie and Hu, Xiaohui and others},
  year   = {2026},
  note   = {TokenWave AI},
  url    = {https://github.com/TokenWave-AI/swe-cowork-bench}
}
SWE-CoWork · TokenWave AI

The future engineer
is an owner.

Human direction. Collective agent execution. A benchmark for the moment when “write the code” becomes “take responsibility for the work.”