Output capture
Extend the result: preserve write order; expose stdout and stderr separately.
- Specwritten up front
- Peoplenone to consult
- Contextall in the ticket
SWE-CoWork places a coding agent inside an organization of digital employees. Requirements sit with accountable colleagues and project records, and each one is tied to executable tests, so we can follow it from the first question to the final handoff.
$ pytest
✓ output checks passed
1result.stdout = out
2result.stderr = err
3result.output = interleaved
Most coding benchmarks hand an agent a packaged issue. SWE-CoWork hands it something closer to a Monday morning: a short onboarding note, a living repository, and a workplace where the missing context belongs to other people.
Extend the result: preserve write order; expose stdout and stderr separately.
The agent receives a brief handoff. It establishes each requirement with the colleague or record that holds it, and then carries that requirement through code, verification, and the final handoff.
There is no prescribed order. New evidence can change the agent’s picture of the work, so a good run keeps circling between people, records, code, and checks.
The agent identifies the target project, scope, and first leads before it spends effort on a patch.
Every assignment is grounded in a real upstream repository and surrounded by an authored project history. Each digital employee has a defined responsibility, bounded local knowledge, and rules for what it can disclose.
Eleven models work against the same 101-task release contract. Switch the metric to compare new behavior delivered (F2P), existing behavior preserved (P2P), and assignments where every test passes in the same run.
Claude-Opus-5.5
GPT-6 Astra
Seed Evolving
Qwen3.8-Max
GLM 5.3
Qwen3.8-Flash
DeepSeek V4.1 Flash
Kimi K3
GLM 5.2
Seed 2.1 Pro
DeepSeek V4 Pro| Model | Messages | Tool calls | Hours | Colleague req. obtained | All req. obtained | F2P |
|---|
Medians per run. Colleague: requirements held only by colleagues that were disclosed. All: adds public requirements, counted as obtained.
F2P does not rise with the amount of interaction. The weakest models, such as Kimi K3 and GLM 5.2, make about 300 tool calls and stop well within the budget, at 24–27% F2P. The middle group, from Seed Evolving to DeepSeek V4.1 Flash, makes 500 to 1,600 tool calls, compacts its context up to 33 times, and obtains up to 59% of the requirements held by colleagues, yet reaches only 28–39%. These agents can already find the relevant colleagues and obtain much of the context, but struggle to keep track of each requirement from clarification through implementation and verification to the delivery report. The two leading models make fewer than 200 tool calls and reach 63–64%, and they share the same limitation on a smaller scale: in Claude-Opus-5.5 runs, 56% of failed nodes with linked requirements fail at implementation, and 31% of requirements reported as done fail their tests.
Claude-Opus-5.5 still uses about twice GPT-6 Astra’s input because of its context window. GPT-6 Astra’s window is about 258K tokens and its median peak context is 245K, so most runs are compacted; Claude-Opus-5.5 has a 1M-token window, a median peak of 483K, and never compacts.
Loading the usage medians…
Usage summaries cover nine of the eleven evaluated models. Token medians are unavailable for Seed 2.1 Pro and DeepSeek V4 Pro. F2P uses the full 101-task release. Dots and short lines mark exact values where nearby logos are offset. M = 1,000,000 tokens.
| Model | Model calls | Input (M) | Output (M) | Tool calls | Compactions | F2P |
|---|
≈ marks approximate input usage. — means unavailable.
How much a model explores does not predict delivery. Claude-Opus-5.5 makes a median of only 56 workplace queries per run, while DeepSeek V4.1 Flash makes 572. The two leading models reach their first code edit at tool calls 67 and 82, still about a third of the way into their short runs. Kimi K3 and GLM 5.2 query and edit on a similar schedule but reach less than half the F2P. The clearest difference is colleague contact: they send medians of 8 and 5 messages, against 19.5 for Claude-Opus-5.5 and 18.5 for GPT-6 Astra. Asking colleagues is not sufficient on its own: GLM 5.3 sends the most messages, 21, and reaches 31.27% F2P.
| Model | Queries before editing | First-edit call | First-edit share | Queries over the run | Messages to colleagues | F2P |
|---|---|---|---|---|---|---|
| Claude-Opus-5.5 | 39 | 82 | 38% | 56 | 19.5 | 63.84% |
| GPT-6 Astra | 52 | 67 | 31% | 101 | 18.5 | 62.91% |
| Seed Evolving | 39.5 | 85 | 12.53% | 134 | — | 39.09% |
| Qwen3.8-Max | 77 | 163 | 23% | 170 | 6 | 35.70% |
| GLM 5.3 | 74 | 227 | 13% | 215 | 21 | 31.27% |
| DeepSeek V4.1 Flash | 272 | 703 | 53% | 572 | 4.5 | 28.20% |
| Kimi K3 | 37.5 | 105 | 37% | 73 | 8 | 27.08% |
| GLM 5.2 | 40.5 | 130 | 34% | 100 | 5 | 24.40% |
First-edit share is the median of each run’s first-edit tool-call index divided by its total tool calls, expressed as a percentage. Query and message counts are medians per run. — means unavailable.
Same model, repositories, tests, scaffold, and budget. The only change is that the complete requirements arrive at the start. With every requirement in hand, Seed Evolving still leaves 44% of F2P nodes unsatisfied and GPT-6 Astra 21%.
Seed Evolving
GPT-6 AstraClaude-Opus-5.5 has the highest F2P score, so its recorded trajectories become a microscope. The traces connect each failed test to its requirement, the colleague who holds it, the question, the patch, and the final handoff note. The agent finds the right people. It loses track of individual requirements afterward.
Most undisclosed requirements belong to owners the agent already contacted: the relevant card was never raised, or the conversation ended without an answer.
42% of failed tests trace back to a requirement that stayed with a colleague. 56% trace back to a requirement the agent received and still did not satisfy.
Dependable delivery means tracking every requirement until it is closed: raise it with its owner, revisit deferred questions, implement it, verify it against the tests, and report its true status in the handoff.
All fourteen figures from the paper, rendered from the source files. Select a figure to enlarge it, or read the full PDF.
@misc{swecowork2026,
title = {SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations},
author = {Lei, Xinping and Liu, Minghao and Zhu, Ruijie and Hu, Xiaohui and others},
year = {2026},
note = {TokenWave AI},
url = {https://github.com/TokenWave-AI/swe-cowork-bench}
}
Human direction. Collective agent execution. A benchmark for the moment when “write the code” becomes “take responsibility for the work.”