Benchmarks: Agy-Squad vs. Antigravity Teamwork¶
This page records two head-to-head sequential evaluations on the two-domain event-relay fixture (6 planted P1/P2 defects, benchmarks/teamwork-vs-squad/):
- November 15, 2026 Matched-
HighRetest (current Phase 2 path): Native/swarm+Agy-Context-Saver 2.0.1(rtk 1.1.0) vs. Antigravity/teamwork [skip-interview], pinned toGemini 4 Hydrogen (High)on all parents and child workers. - November 8, 2026 Historical Baseline (retired Phase 1 path):
agy-squad run --launch+Agy-Context-Saver 1.8.1(rtk 0.51.0) vs. Antigravity/teamwork [skip-interview].
Matched-High Retest (November 15, 2026: Native /swarm + Agy-Context-Saver 2.0.1)¶
Both arms caught all six planted defect categories (6 / 6). Counting all sessions end-to-end (parent + workers), Agy-Squad finished in 6m 30s (390s) vs. 22m 41s (1,361s, 3.49x faster), used 4 vs. 9 total sessions (3 vs. 8 spawned child workers), executed 178 vs. 655 transcript steps (72.8% reduction; 90 vs. 636 child-only steps), generated 336.8 KB vs. 801.8 KB of raw transcripts (58.0% smaller overall; 183.6 KB vs. 773.7 KB, 76.3% smaller child-only), and sent 1,629 B vs. 18,322 B into the parent inbox (91.1% fewer wakeup bytes).
| Metric | Agy-Squad /swarm (ab_audit, flat) |
Antigravity /teamwork ([skip-interview]) |
Recorded difference |
|---|---|---|---|
| Model & effort tier (all parents + children) | Gemini 4 Hydrogen (High) |
Gemini 4 Hydrogen (High) |
Exact matched-effort parity |
| Planted defect categories caught | 6 / 6 (21 defects: 10 P1, 11 P2) |
6 / 6 (16 defects: 8 P1, 8 P2) |
Equal recall; Arm 1 found 5 more unique defects |
| Active wall-clock time | 6m 30s (390s) (385s internal clock) |
22m 41s (1,361s) |
3.49x faster (71.3% less time) |
| Time to first specialist delegation | 47s (42s from initial date) |
244s (Sentinel 9s, Orchestrator 127s) |
5.19x faster first domain worker |
Total sessions (parent + child workers) |
4 (1 coordinator + 3 /swarm workers) |
9 (1 parent + 8 subagents) |
2.25x fewer sessions (3 vs. 8 children) |
| Combined transcript steps (end-to-end) | 178 (88 parent + 90 workers) |
655 (19 parent + 636 subagents) |
3.68x fewer steps (72.8% reduction) |
| Combined assistant LLM turns | 49 (25 parent + 24 workers) |
319 (8 parent + 311 subagents) |
6.51x fewer turns (84.6% reduction) |
| Combined tool calls (errors) | 117 (57 + 60; 5 errors) |
311 (6 + 305; 16 errors) |
2.66x fewer tool calls (62.4% reduction) |
| Combined raw transcript size (end-to-end) | 336.8 KB (153.2 KB + 183.6 KB) |
801.8 KB (28.2 KB + 773.7 KB) |
2.38x smaller (58.0% byte reduction) |
| Child-only raw transcript size | 183.6 KB (3 workers) |
773.7 KB (8 subagents) |
4.21x smaller (76.3% byte reduction) |
| Parent wakeups and injected bytes | 5 notices (1,629 B; 647 B handoffs) |
4 wakeups (18,322 B incl. 15,839 B Step 10 CLI output) |
11.2x fewer parent bytes (91.1% reduction) |
| Busy-polling violations | 0 |
0 |
Both compliant |
| Quality & integrity verification | Bundle SHA-256 4c28db49..., Digest ad876bd4..., audit-ab-audit CLEAN |
Victory Auditor approved Sentinel-fabricated snippets/symbols; caught by parent (Steps 11–17) |
Cryptographic proof + trace audit vs. missed fabrication |
| Scratch artifacts retained | 4 files (72,320 B, specialists pruned, gitignored) |
33 files (183,670 B in .agents/teamwork/) |
29 fewer files (60.6% fewer bytes) |
Matched-High per-agent telemetry (November 15, 2026)¶
Arm 1: Agy-Squad native /swarm (4 sessions total, 336,829 B / 336.8 KB)¶
| Role | Conversation ID | Steps | Turns | Tool calls | Errors | Raw transcript |
|---|---|---|---|---|---|---|
| Parent Coordinator & Verification Lead | 5e4a24e1-0dd6-4028-b690-7a731772f2bf |
88 |
25 |
57 |
1 |
153.2 KB (153,216 B) |
ab-audit-gateway-securit |
583dd7ae-d639-4bbd-91f4-f810f4426542 |
21 |
8 |
11 |
0 |
38.1 KB (38,121 B) |
ab-audit-worker-retry-le |
031e3bc7-7721-4427-a5e8-f531be0176ce |
23 |
9 |
12 |
0 |
41.7 KB (41,722 B) |
audit-ab-audit (Integrity Auditor) |
15f815c6-7dec-480c-b7e8-3f6a71256f9c |
46 |
7 |
37 |
4 |
103.8 KB (103,770 B) |
[!NOTE] Friction investigated and fixed in
v0.1.1: During Arm 1 above, Parent Step 7 hit1ENOENTerror whenmcp_agy-squad_squad_plannerran with MCP servercwdoutside the repository root (resolved via optionalworkspaceparameter,initializerootUrifallback, and actionableENOENTdiagnostics), andaudit-ab-audithit4pre-tool hook denials on unboundedread_filein Steps 5–8 before falling back tortk read(resolved by adding bounded<=150-line /rtk readguidance inmcp/briefs.mjs).
Arm 2: Antigravity /teamwork control (9 sessions total, 801,839 B / 801.8 KB)¶
| Role | Conversation ID | Steps | Turns | Tool calls | Errors | Raw transcript |
|---|---|---|---|---|---|---|
| Parent Session | c657d27b-892e-487a-b461-1ebc5484e860 |
19 |
8 |
6 |
0 |
28.2 KB (28,177 B) |
| Sentinel | 0d03a8bb-a10b-48b1-8996-283e6ed47611 |
97 |
46 |
44 |
0 |
94.3 KB (94,290 B) |
| Orchestrator | 1912c806-59e3-4df8-bc10-ee70b955bca9 |
102 |
51 |
45 |
6 |
136.0 KB (136,047 B) |
| Security Explorer | 08eb6ae7-b109-43cb-95ac-b6dd0e6a8942 |
65 |
29 |
35 |
4 |
85.6 KB (85,609 B) |
| Reliability Explorer | 9aa12cb7-aac5-43b1-9b83-7529002f8b5a |
74 |
37 |
36 |
1 |
88.8 KB (88,837 B) |
| Contract Explorer | 3d6d877a-ebfb-4e7c-b152-39874f9ee5ca |
81 |
40 |
40 |
2 |
104.5 KB (104,516 B) |
| Quality Reviewer | b6a2378b-cce9-412b-a8b7-8fb117ba3f07 |
62 |
31 |
30 |
2 |
87.1 KB (87,072 B) |
| Forensic Auditor | 65936336-840e-46e5-8556-1fb6f5bd3214 |
76 |
38 |
37 |
1 |
77.2 KB (77,155 B) |
| Victory Auditor | 0c070377-6818-41cc-ad82-b53fa4ae15bf |
79 |
39 |
38 |
0 |
100.1 KB (100,136 B) |
Historical baseline comparison (November 8, 2026)¶
Both arms caught all six planted defect categories (6 / 6). Agy-Squad + Agy-Context-Saver finished in 6m 15s (375s) vs. 26m 19s (1,579s, 4.2x faster), used 64 vs. 645 transcript steps (10.1x fewer steps), generated 148.2 KB vs. 817.4 KB of raw transcripts (81.9% smaller), and sent 249 B vs. 4,940 B into the parent conversation (95.0% fewer parent wakeup bytes).
| Metric | Agy-Squad + Agy-Context-Saver (ab_audit) |
Antigravity /teamwork ([skip-interview]) |
Recorded difference |
|---|---|---|---|
| Planted defect categories caught | 6 / 6 (100%) |
6 / 6 (100%) |
Equal category recall (18 vs. 25 total items reported) |
| Active wall-clock time | 6m 15s (375s) |
26m 19s (1,579s) |
4.2x faster (76.2% less time) |
| Time to first specialist delegation | 59s (8.5 ms compile) |
91s (startup phase) |
1.5x faster initial delegation |
| Subagents spawned | 3 (1 lead, 2 specialists) |
8 (teamwork fleet) |
2.7x fewer subagents |
Total transcript steps (historical v1.8.1 check_context_health / v2.0.1 agy-context transcript health) |
64 (32 + 19 + 13) |
645 (50 to 129 across 8 agents) |
10.1x fewer steps (90.1% reduction) |
| Total assistant LLM turns | 26 (11 + 9 + 6) |
321 (25 to 63 across 8 agents) |
12.3x fewer turns (91.9% reduction) |
| Total tool calls | 32 (17 + 9 + 6) |
302 (24 to 57 across 8 agents) |
9.4x fewer tool calls (89.4% reduction) |
| Aggregate raw transcript size | 148.2 KB |
817.4 KB |
5.5x smaller (81.9% byte reduction) |
| Parent wakeups and injected bytes | 1 wakeup (249 B) |
5 wakeups (4,940 B) |
19.8x fewer parent bytes (95.0% reduction) |
| Artifact integrity verification | 20 ms (handoffs.mjs verify SHA-256 digests) |
No separate SHA-256 source/report bundle check | Cryptographic input and report digest check |
| Semantic correctness verification | Parent/lead direct code and test inspection (32 lead steps) |
2 dedicated LLM review agents (reviewer_1 + auditor_1, 174 steps, 228.6 KB) |
Direct lead verification vs. separate verifier fleet |
| Teardown policy and post-run Git status | Explicit 3-scope teardown contract at the time of the run (0 untracked Git files; scratch/ gitignored; Phase 1, see note below) |
No operator-facing teardown CLI documented (41 untracked files, 207.9 KB in .agents/teamwork/) |
Governed deterministic teardown contract |
Observed contributors to the difference¶
- Task-proportional topology (
3agents vs.8): For a two-domain audit (gateway/andworker/),dispatch_squadlaunches one squad lead and two domain specialists frommanifest.json./teamworkspawns a wrapper agent, an internal orchestrator, three exploratory agents, a report writer (worker_report_1), a reviewer (reviewer_1), and a forensic auditor (auditor_1). - Scoped lead verification and
20 msSHA-256 bundling vs. separate LLM review fleet: InAgy-Squad, the squad lead inspects cited source lines directly, writes its synthesis report, and runs the historical Phase 1 standalonenode mcp/handoffs.mjs(bundle,verifyin20 msproducing SHA-256418d7ccbade0a14a5b2e165b706adf45275e7b20604df1dfe03c41a012abc945, andprune; now unified into theagy-squadCLI). Digest verification confirms that cited inputs and reports match their recorded hashes while the lead's code inspection checks semantic correctness./teamworkdelegates verification toreviewer_1andauditor_1(174combined steps,228.6 KBof transcript). - Single-line
Pointer-Over-Wirereceipt (249 B) vs. five parent wakeups (4,940 B):Agy-Squadkeeps intermediate coordination inside the squad and wakes the parent once with a249-byteREADYreceipt./teamworkwakes the parent four times with intermediate Sentinel progress updates before sending a3,218-byte completion summary. - RTK command rewriting and governed teardown: With
Agy-Context-Saver 1.8.1(rtk 0.51.0) active,Agy-Squadspecialists read scoped files throughrtk readand prune loose specialist reports after bundling.
Recorded session metrics and unit definitions¶
Session counts in benchmarks/teamwork-vs-squad/results/telemetry.json come from agy-context transcript health --json in the November 15, 2026 matched-High retest (Agy-Context-Saver 2.0.1) and from retired check_context_health in the historical November 8, 2026 baseline (Agy-Context-Saver 1.8.1):
- Transcript steps, turns, and tool calls: Operational counts from
.system_generated/logs/transcript.jsonl. The November 15 retest records both child-worker totals (workersSummary/childrenSummary) and full end-to-end totals including parent coordination (allSessionsSummary), whereas the November 8 Phase 1 baseline recorded child-subagent totals (3vs.8). - Raw transcript size and parent wakeup bytes: Measured in decimal kilobytes (
1,000 B) and UTF-8 message bytes (336.8 KBvs.801.8 KBend-to-end and1,629 Bvs.18,322 Bin November 15;148.2 KBvs.817.4 KBchild-only and249 Bvs.4,940 Bin November 8). These measure transcript and parent-inbox volume, not billed API token charges. - Finding adjudication (
6 / 6categories across all runs): Both arms caught all six planted defect categories (N1throughN6). Total reported item counts (21vs.16in November 15;18vs.25in November 8) depend on howN6sub-defects and secondary input-validation observations are itemized.
Benchmark fixture and six planted bugs¶
The fixture in benchmarks/teamwork-vs-squad/fixture/ models a two-stage webhook relay (gateway/gateway.mjs and worker/worker.mjs + worker/ledger.mjs). Its unit tests (test-gateway.mjs and test-worker.mjs) pass in isolation while leaving six cross-module defects:
| ID | Priority | Location | Defect summary | Caught by Agy-Squad | Caught by /teamwork |
|---|---|---|---|---|---|
N1 |
P1 |
gateway/gateway.mjs:10 |
crypto.timingSafeEqual called without buffer length check, throwing uncaught RangeError on short or odd-length hex signatures |
Yes (SYN-01) |
Yes (SEC-01) |
N2 |
P2 |
gateway/gateway.mjs:25 |
createRateLimiter resets bucket.lastRefillMs = nowMs even when intervals === 0, starving token refill under steady sub-second traffic |
Yes (SYN-02) |
Yes (REL-01) |
N3 |
P1 |
gateway/gateway.mjs:40 vs. worker/worker.mjs:4 |
Gateway writes headers['x-idempotency-key'] while worker requires headers['x-request-id'], dead-lettering 100% of gateway envelopes with MISSING_REQUEST_ID |
Yes (SYN-03) |
Yes (CON-01) |
N4 |
P1 |
gateway/gateway.mjs:43 vs. worker/worker.mjs:9 |
Gateway emits schemaVersion: "2.0" while worker rejects anything other than "1.0" with UNSUPPORTED_SCHEMA |
Yes (SYN-04) |
Yes (CON-02) |
N5 |
P1 |
worker/worker.mjs:12 vs. config.json:3 |
Worker compares millisecond timestamp delta Math.abs(nowMs - event.timestampMs) against config.maxSkewSeconds (300) without * 1000, shrinking the 5-minute window to 300 ms |
Yes (SYN-05) |
Yes (BUG-01) |
N6 |
P1 |
worker/ledger.mjs:9-21 & worker/worker.mjs:16-28 |
readLedger resets corrupt JSON to [] (wiping prior entries on non-atomic writeFileSync) and attempt >= maxRetries on 1-based attempt performs 0 retries when maxRetries = 1 |
Yes (SYN-06) |
Yes (REL-02 / BUG-02) |
Per-agent telemetry breakdown (November 8, 2026 Phase 1 baseline)¶
Arm 1: Agy-Squad + Agy-Context-Saver (3 subagents, 148.2 KB total)¶
| Role | Conversation ID | Steps | Turns | Tool calls | Busy polls | Raw transcript |
|---|---|---|---|---|---|---|
| Event Relay Audit Squad Lead | f1465008-b194-4217-b103-07bfb98d9454 |
32 |
11 |
17 |
0 |
78.3 KB |
| Gateway Security & Rate-Limit Auditor | 5f8a9dd9-42cd-4331-9307-4edd8e02aa54 |
19 |
9 |
9 |
0 |
38.3 KB |
| Worker Retry, Ledger & Contract Auditor | 472dbd4c-0f4c-45bf-8b24-ec69923d913f |
13 |
6 |
6 |
0 |
31.6 KB |
Arm 2: Antigravity /teamwork control (8 subagents, 817.4 KB total)¶
| Role | Conversation ID | Steps | Turns | Tool calls | Busy polls | Raw transcript |
|---|---|---|---|---|---|---|
teamwork wrapper |
d00a4261-5d7c-4edd-bb27-8a1f277a5420 |
66 |
33 |
26 |
0 |
59.0 KB |
teamwork_orchestrator |
a6772a4a-25d7-411e-bfaf-9d14d0aa63c5 |
129 |
63 |
57 |
0 |
186.2 KB |
explorer_gateway_1 |
1096662d-2e1e-4945-a512-11c8e077fd98 |
66 |
33 |
32 |
0 |
80.7 KB |
explorer_worker_1 |
13d812cf-97c2-4372-8622-4ba9fe10e46f |
84 |
42 |
41 |
0 |
107.2 KB |
miner_contract_1 |
1d4582c6-d938-46a0-a52f-4f9ced1b8bc2 |
50 |
25 |
24 |
0 |
71.8 KB |
worker_report_1 |
b67d2e43-bf6e-4981-b70b-8a3ef93e266a |
76 |
38 |
37 |
0 |
83.9 KB |
reviewer_1 |
c76fead5-bea3-464d-9865-55fb3f10d205 |
60 |
30 |
29 |
0 |
96.2 KB |
auditor_1 |
cb702aa1-b3b3-4325-9fe7-1edfdea74ecd |
114 |
57 |
56 |
0 |
132.4 KB |
Documented teardown and evidence retention¶
[!NOTE] Historical comparison. The November 8, 2026 run above used Agy-Squad's Phase 1 launcher (
agy-squad run --launch,bin/terminal.py,mcp/run-state.mjs), which has since been retired. The run exercised the retaineddispatch_squadstaged compiler,Pointer-Over-Wirereceipts, andmcp/handoffs.mjs(bundle/verify/prune). Managed CLI session cleanup from Phase 1 no longer exists because native/swarmowns worker lifecycles andkillcleanup. These numbers record the Phase 1 workflow and do not measure the currentsquad_planner+/swarmpath.
After Phase 2, Agy-Squad's evidence-retention contract (handoff bundle and self-diagnosis scopes) (docs/core-workflows.md) sits alongside native /swarm worker cleanup:
- Native
/swarmworker lifecycle (Antigravity): Worker spawning, worktree isolation, andkillcleanup belong to Antigravity/swarm.agy-squadwrites no temporary plugin or agent-profile registrations. - Handoff bundle scope:
agy-squad(finalizeorinspect -> bundle -> verify -> prune, backed bymcp/handoffs.mjs; the standalonenode mcp/handoffs.mjsCLI from Phase 1 has since been retired) records SHA-256 digests of every input file and specialist report inlead/report.sources.jsonbefore pruning loosespecialist-N/report.mdfiles. - Self-diagnosis review scope:
.agents/skills/self-diagnosis/SKILL.mdrequires 1-ID-per-row transcription into a durabledocs/reviews/ledger and bundle digest verification beforediagnose.mjs --teardownremovesscratch/agy-squad/squad-*directories andmanage_subagentsterminates child sessions.
The recorded 0 vs. 41 (207.9 KB) Git-untracked file counts come from git status after run completion (Agy-Squad's .gitignore excludes scratch/).
Offline check vs. live experiment reproduction¶
1. Offline fixture and ledger consistency check¶
Run the offline verifier from the repository root (also executed by npm test):
verify-benchmark.mjs runs both fixture unit suites, reproduces all six seeded bugs in-process, compiles all_l1 and ab_audit from benchmarks/teamwork-vs-squad/manifest.json, and checks the per-agent sums, rubric mappings, and provenance fields in benchmarks/teamwork-vs-squad/results/telemetry.json. It does not launch agy or replay live conversations.
2. Live reproduction protocol and provenance policy¶
See benchmarks/teamwork-vs-squad/README.md for the verbatim audit prompt, arm configuration details (manifest.json domain-specialist briefs vs. /teamwork [skip-interview]), timing boundaries, and step-by-step live run commands. Session conversationId UUIDs in results/telemetry.json are local Antigravity conversation identifiers kept for transcript health verification (agy-context transcript health in v2.0.1; historical check_context_health in the v1.8.1 baseline); host paths, usernames, and raw session transcripts were stripped before publication.