Reproduce this benchmark

Tools were installed from release binaries, so we record release tags and build ids; we did not record source commits for them.

OS

Debian GNU/Linux 13 (trixie), kernel 6.12, x86_64

Hardware

8 vCPU, 16 GB RAM, no GPU, no Docker

Display

Xvfb virtual display for GUI apps and orca serve

Accounts

No authenticated agent CLI (Claude, Codex, Cursor, Grok). Nothing bought, no sign-ups.

GitHub snapshot

data/gh_snapshot.json, 2026-10-19 17:25 UTC

• t1 Bug fix: in shelf/paging.py, paginate() is documented as 1-indexed but returns wrong items, and page_count() drops the final partial page. Fix both bugs. page 0 or negative pages must return []. Do not touch other files. Run the existing tests with: python3 -m unittest discover -s tests -t .
• t2 Feature: add a new module shelf/slug.py with slugify(text, max_len=None). Lowercase, replace any run of non-alphanumeric characters with a single '-', strip leading/trailing '-'. If max_len is given, truncate to max_len characters then strip any trailing '-'. slugify('???') returns ''. Only create/modify shelf/slug.py.
• t3 Refactor: shelf/report.py has duplicated formatting logic in book_line and loan_line. Extract one shared helper (e.g. _fmt_row(title, who, year)) so the column formatting (.ljust(30)[:30], .ljust(20)[:20], rjust(4)) appears once. Public function names and output must stay byte-for-byte identical. Existing tests must still pass: python3 -m unittest discover -s tests -t .
• t4 Write tests: shelf/stats.py has no tests. Create tests/test_stats.py using unittest covering mean, median (odd and even length, unsorted input), clamp (below, inside, above, lo>hi error) and empty-input errors. Do not modify shelf/stats.py. Tests must pass: python3 -m unittest discover -s tests -t .
• t5a Change: in shelf/config.py make the default 'retries' value 5 instead of 3. Update tests/test_smoke.py if needed.
• t5b Feature: in shelf/config.py add a keyword argument env="dev" to load_config(overrides=None, env="dev"); the returned dict must include key 'env' set to that value (overrides may still override it).

fixture/ holds the repo, tasks.json, the hidden verifiers and reference solutions. The stub agent is harness/benchagent.py.

1. Clone and check the fixture

   git clone https://github.com/arjunkshah12345-hash/agent-orchestrator-bench.git && cd agent-orchestrator-bench
   # verifiers must fail on the base repo and pass on the references
   for t in t1 t2 t3 t4 t5a t5b; do python3 harness/verify.py fixture/repo $t; done

2. Baseline (plain git worktree + stub agent)

   python3 harness/baseline_run.py A-baseline-stub 'env STUB_SLEEP=8 python3 harness/benchagent.py {prompt}'

3. Install the orchestrators

   # Orca v1.4.230 AppImage from github.com/stablyai/orca/releases, extract to tools/orca-x
   ./orca-linux.AppImage --appimage-extract && mv squashfs-root tools/orca-x/
   # Claude Squad v1.0.20 from github.com/smtg-ai/claude-squad/releases into tools/cs/claude-squad

4. Orca, 3 runs

   Xvfb :90 &
   HOME=/tmp/home-orca-cli DISPLAY=:90 tools/orca-x/squashfs-root/resources/bin/orca-ide serve &
   python3 harness/orca_run.py   # repeat 3x; copy runs/F-orca/result.json + resources.csv per run
   STUB_CRASH=t3 python3 harness/orca_run.py   # crash test

5. Claude Squad, 3 runs

   python3 harness/cs_run.py   # drives the TUI through tmux; repeat 3x
   STUB_CRASH=t3 python3 harness/cs_run.py

6. Merge check

   python3 harness/merge_eval.py   # expects 5/6 clean, t5b conflicting in shelf/config.py

7. GUI launch probes

   python3 harness/gui_probe.py <name> <path-to-AppImage>   # first window time + RSS on Xvfb

8. Rebuild the site from the logs

   python3 site/build.py && python3 -m http.server -d site 8000

• runs/repeat/ Orca and Claude Squad, 3 runs each: run JSON + 1 s resource CSVs
• runs/F-orca/ Orca crash test, merge, serve readiness and restarts
• runs/C-claude-squad/ Claude Squad screens, crash test, tmux-kill recovery, merge
• runs/A-baseline-stub/ plain worktree baseline
• runs/E-gui-*/ GUI launch probes with screenshots
• runs/D-t3code/ T3 Code server probe
• runs/B*/ Lane B real-model attempts (all failed, not scored)
• runs/R-real-round-DRYRUN-stub/ real-agent harness plumbing check with the stub agent (not a result)
• matrix.csv full feature matrix and assessment data
• report/tools_data.py rubric, criterion scores and rationale
• data/gh_snapshot.json GitHub API snapshot

3 orchestrators (plain git worktree as control, Orca, Claude Squad) x 1 model x 3 tasks (t1 bug fix, t2 feature, t3 refactor) x 3 repetitions. Each cell records verified success from the hidden verifiers, agent exit, wall time, tokens and peak RSS. A plumbing check with the stub agent passed all 27 cells; that only shows the harness works and is not a result.

python3 harness/real/real_round.py \
  --agent-cmd 'claude -p {prompt} --model {model} --output-format json --dangerously-skip-permissions' \
  --model <one model id> --tools baseline,orca,cs --tasks t1,t2,t3 --reps 3

What it needs

• One authenticated agent CLI on the box, used for all 27 cells. For example Claude Code logged in with claude login (Claude subscription) or ANTHROPIC_API_KEY set; or Codex CLI with codex login / OPENAI_API_KEY; or opencode with a provider key whose rate limit fits one request (the Groq free tier at 7 to 8k tokens per minute does not: one opencode request was 7.3 to 8.6k tokens).
• orca serve running on an Xvfb display (step 4) and tmux for Claude Squad.
• Budget for 27 agent runs (3 tools x 3 tasks x 3 reps). Tokens are taken from the CLI's own JSON usage output; if a CLI prints none, tokens are recorded as empty, never estimated.