How the scores were made, and what they are not
This is a feature and orchestration benchmark, not a model leaderboard. The ranking is a weighted feature assessment, not measured performance. Scores are editorial judgement on a 0 to 5 scale per criterion, backed by vendor docs, repos, a GitHub snapshot (2026-10-19 17:25 UTC) and hands-on probes on one Linux box. There are no real-model task results in any score: no Claude, Codex, Cursor or Grok agent CLI was logged in on the box.
Nine criteria, 100 points of weight
| Key | Criterion | Weight | What a 5 means |
|---|---|---|---|
| A | Agent breadth | 15 | 5: 8+ agent CLIs documented, or arbitrary CLI proven |
| B | Isolation and parallelism | 15 | 5: strong per-task isolation (worktree+remote, or containers) plus fan-out |
| C | Review, merge and PR flow | 15 | 5: in-app diff review plus commit/PR creation and PR lifecycle |
| D | Remote, cloud and mobile reach | 10 | 5: phone/web/remote runtimes or cloud execution |
| E | Openness and license | 10 | MIT/Apache 5, AGPL 4, source-available 3, closed 1 |
| F | Maturity and momentum | 15 | stars, release cadence/recency, vendor status, open-issue load |
| G | Linux setup friction (measured) | 10 | from our hands-on: install, account walls, launch, footprint |
| H | Pricing and BYO model | 5 | 5: free, bring your own subscription |
| I | Docs and transparency | 5 | docs site, architecture docs, honest status notes |
Score = sum(weight x criterion score) / 100 x 20. A criterion with no evidence is unranked (shown as n/r). Each tool gets a range: the minimum treats unranked criteria as 0, the maximum as 5. The point score is the midpoint of that range; read it together with the range and the doc coverage, hands-on validation and measured-results columns.
Test environment
- Machine
- Linux x86_64, 8 vCPU, 16 GB, no Docker.
- Accounts
- No authenticated agent CLIs. We did not buy credits or sign up for anything.
- Fixture
- A small Python repo with 6 tasks: bug fix, feature, refactor, test-writing checked against seeded mutants, and two deliberately conflicting edits (t5a, t5b). Hidden verifiers decide pass or fail.
- Verifier self-test
- All verifiers fail on the base repo, all pass on reference solutions, and t5a + t5b really conflict.
Orchestrator mechanics with a stub agent
A deterministic stub agent stands in for a model so we can measure the orchestrator itself: parallel launch, isolation, crash behaviour and merge. It says nothing about how well a real model codes inside each tool.
| Harness | Runs | All launched (s) | Mean wall (s) | Verified | Peak RSS MB |
|---|---|---|---|---|---|
| Baseline git worktree | 1 | 0.043 | 8.4 | 6/6 | - |
| Claude Squad | 3 | 11.18 to 11.89 | 24.6 | 18/18 | 102 to 118 |
| Orca (orca serve CLI) | 3 | 1.52 to 1.90 | 16.9 | 18/18 | 1124 to 1140 |
Time to first window on Xvfb
| Tool | First window (s) | RSS MB |
|---|---|---|
| Cloudroom | 3.3 | 795 |
| Paseo | 4.65 | 852 |
| Orca | 4.82 | 920 |
| OpenChamber | 5.32 | 1297 |
| Mux (now 'Xum') | 6.08 | 1065 |
| Emdash | 6.59 | 3932 |
| Synara | 8.37 | 2134 |
| Superset | 11.41 | 895 |
| T3 Code server | 1.01 (port) | 394 |
What we could not test
- No real-model task success. No Claude, Codex, Cursor or Grok CLI was logged in. The only key was a Groq free tier (7 to 8k tokens per minute) and one opencode request was 7.3 to 8.6k tokens. 0 of 9 attempts passed (3 rate-limit, 153 too-large and 3 tool-schema errors in the logs). Those attempts are shown below for transparency and are not used in any score.
- T3 Code and Synara sessions. Both orchestrate agent CLIs that must be installed and authenticated; none were.
- Cloudroom core. Its installer needs root and systemd on a fresh Ubuntu 24.04 VM, which we did not do on a shared box.
- Closed or macOS-only tools (Conductor and others) were scored from docs and secondary sources only.
- Judgement. Criterion scores are one reviewer's reading of the evidence. Use the sensitivity table, the score range and the coverage columns to see how much a ranking depends on it.
- Snapshot. Stars, releases and pricing are as of 2026-10-19 and several tools ship daily.
| Model | Task | Exit | Wall s | Verified |
|---|---|---|---|---|
| groq/openai/gpt-oss-120b (6 parallel) | t4 | 1 | 179.5 | fail |
| groq/openai/gpt-oss-120b (6 parallel) | t2 | 1 | 233.3 | fail |
| groq/openai/gpt-oss-120b (6 parallel) | t5b | 1 | 234.8 | fail |
| groq/openai/gpt-oss-120b (6 parallel) | t5a | 1 | 292.2 | fail |
| groq/openai/gpt-oss-120b (6 parallel) | t1 | 1 | 292.5 | fail |
| groq/openai/gpt-oss-120b (6 parallel) | t3 | 1 | 347.9 | fail |
| groq/openai/gpt-oss-20b (serial) | t1 | 1 | 131.7 | fail |
| groq/openai/gpt-oss-20b (serial) | t2 | timeout | 400.0 | fail |
| groq/qwen/qwen3.8-27b (serial) | t1 | aborted | - | fail |