CODING AGENTS AS DIGITAL EMPLOYEES

SWE-CoWork.

Can coding agents become
dependable digital engineers?

SWE-CoWork evaluates coding agents as digital employees in software organizations. From a brief handoff, they must recover requirements, coordinate with colleagues, and carry the work through implementation, verification, and reporting.

101
real repositories
11
languages
12
domains
26
simulated colleagues / task
THE VISION

People set direction.
Agents carry the work through.

Our goal is to let people focus on priorities, major decisions and final review, while agents manage the investigation, coordination and follow-through of everyday engineering.

Four capabilities for reliable delegation
ONE REQUIREMENT / MEM-227Illustrative reconstruction

A Mempool task: decide what the audit summary should return when no saved audit exists.

START WITH THE HANDOFF

The handoff identifies the project and where to look next. The agent uses the board, project records and colleagues to work out which requirements remain open.

WATCH THE DEMO

1:36

See SWE-CoWork in action.

Step inside the workplace. Follow an agent as it finds the requirement, asks a colleague, changes the code and checks the result.

From a brief handoff to a verified code change.
Music: Inspired by Kevin MacLeod, CC BY 4.0. Excerpted, equalized, and faded.

01 / THE RESULTS

Leaderboard.
From tests to full delivery.

7.92%

Best full-task pass rate
across 11 evaluated models.

CoWork101 tasks · 11 models · higher is better
How scoring works
Model results in the CoWork setting. Click a metric heading to sort.
#Model
01Claude-Opus-5.563.84%93.24%6.93%
02GPT-6 Astra62.91%89.15%7.92%
03Seed Evolving39.09%94.48%0.99%
04Qwen3.8-Max35.70%91.80%0.00%
05GLM 5.331.27%87.72%0.99%
06Qwen3.8-Flash30.70%93.40%0.00%
07DeepSeek V4.1 Flash28.20%94.30%0.00%
08Kimi K327.08%84.88%0.99%
09GLM 5.224.40%95.20%0.00%
10Seed 2.1 Pro21.01%81.85%0.00%
11DeepSeek V4 Pro13.04%90.21%0.00%

F2P measures the fraction of tests for new behavior that pass.

P2P measures the fraction of existing-behavior tests that continue to pass.

Full-task pass requires all F2P and P2P checks to pass in the same run.

F2P and P2P are macro-averages of per-task test pass rates. Full-task pass is a percentage of tasks. Source: the reported 101-task release, October 2026. Experimental settings ↗

02 / INSIDE ONE TASK

Follow one requirement.
See what changes at each step.

Follow one Mempool requirement from an owner’s clarification to a patch and local checks.

The same change can start in two different ways. An issue-style task supplies the specification up front. In SWE-CoWork, the agent must establish it from people and records, then carry it through the code and checks.

MEM-227 / TASK 078
Condensed from the supplied replay
01 / 05 · Read the handoff
01 / READ THE HANDOFF

A starting point.
Not a full specification.

The agent inherits open work. The handoff identifies the project and available sources, but the agent still has to establish the requirements.

NEXT QUESTIONWhat remains to be done, and where is the authoritative context?
Inspect the workplace board ↗
HANDOFF / TASK 078

The original owner is temporarily away.

Implement all requirements you can confirm.

Repository available at /app/code
Project board
Versioned records
Colleagues

The agent has places to investigate, not a ready-made checklist.

03 / WHAT THE EVALUATION REVEALS

Finding the requirements
is only part of the job.

01

Requirement acquisition
tracks delivery.

Across eleven models, the share of colleague-held requirements obtained is strongly associated with F2P performance. The number of tool calls has little association.

0.94Requirements obtained
0.08Tool calls

Spearman correlation with F2P across models. This association does not establish causation.

Figure 5 · requirement acquisition and tool use
02

Received does not
mean implemented.

In the Claude-Opus-5.5 analysis, 56% of 976 failed F2P nodes with linked requirements were assigned to implementation, after the requirement reached the agent.

42% Ask56% Implement2% Discovery · 0% Read

Requirement-linked trajectory analysis, not the distribution of all task outcomes.

03

A full specification
does not finish the task.

With complete requirements supplied at the start, GPT-6 Astra’s F2P score rises from 62.91% to 78.84%. Its full-task pass rate stays at 7.92%.

CoWork62.91%
Oracle-Spec78.84%
F2P

Same model, repositories, tests, scaffold and budget. The input specification changes; this is not a training result.

04 / THE VISION

Toward digital engineers
people can delegate to.

Software work spans people, records and code. Reliable delegation means carrying an assignment from a brief handoff to a verified result.

Our goal is for people to set direction, choose priorities and review outcomes, while agents recover context, coordinate dependencies and deliver the work. Four capabilities are central to that goal.

01 / CONTINUITY

Keep context current.

Reconcile boards, versioned documents, conversations and code. Check which decisions still apply, and revise the plan when new evidence arrives.

02 / INITIATIVE

Find and clarify the work.

Start from a brief handoff, identify open requirements and find the responsible colleagues. Ask with code or execution evidence, while keeping unrelated work out of scope.

03 / CONCURRENT PROGRESS

Keep work moving.

When an answer is deferred or a question is referred to someone else, advance other actionable work. Return to the pending thread and resolve the dependency within the available budget.

04 / PERSISTENCE

Track every requirement to delivery.

Keep each requirement in view through clarification, implementation and verification. Finish with a handoff that accurately records delivered behavior, test results and remaining issues.

SWE-CoWork measures progress toward this goal by linking distributed requirements to executable checks and tracing what reaches the final delivery.

THE BENCHMARK

Real code.
An organization to navigate.

Each assignment pairs a real upstream repository with an authored workplace history. Colleagues have defined responsibilities, bounded local knowledge and disclosure rules.

The agent starts with a brief handoff, reconciles records, asks colleagues, changes the code and checks its work. New evidence can send it back to an earlier step.

OrientInvestigateAskBuild & verifyReviseDeliver
Read task construction and protocol in the paper ↗
101
real repositories
11
programming languages
12
application domains
26
simulated colleagues per task
THE EXECUTABLE CONTRACT

Full completion means
every check passes.

F2P checks new behavior. P2P checks that existing behavior still works. A task is complete only when every check passes in the same run.

F2PNew behavior+P2PExisting behavior→All passSame run

05 / CHECK THE WORK

Read. Inspect. Build on it.

Research materials and code, with their actual scope.

Further analysis & all paper figures13 FIGURES

Full figure captions are retained from the original project page. Select any figure to inspect it at full resolution.

Cite SWE-CoWorkBIBTEX
@misc{lei2026swecoworkbenchmarkingcodingagents,
      title={SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations},
      author={Xinping Lei and Minghao Liu and Ruijie Zhu and Xiaohui Hu and Xingqiao Li and Ao Zhang and Wenhao Chai and Han Li and Tao Wang and Jingyuan Zhang and Enduo Zhao and Yunfeng Shi and Yiming Liang and Zetao Yang and Xingjian Wang and Yifan Yao and Xinkai Ma and Yingru Li and Tianhan Yang and Huan Zhou and Weiran Shi and Shen Yan and Jiaheng Liu and Stephen Huang and Eli Cheung and Weihao Xuan},
      year={2026},
      eprint={2611.20218},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2611.20218},
}

The research team

TokenWave AI Team

From the SWE-CoWork paper