PR #24 Workspace editing benchmark¶

On 2026-11-05, 18 Antigravity trials compared native file editing, structural editing with host lifecycle orchestration, and agent-owned Workspace import, guarded editing and SAVE publication. Each arm ran three times in Git and three times in non-Git against the same five-file, 21-edit task.

Workspace (observed_workspace) averaged 296.742 seconds from timed start through verification, compared with 354.234 seconds for Native, a 16.2% reduction. It used 32.00 editor calls per trial versus 55.83, a 42.7% reduction. All arms produced exact output. Workspace handled initialization and publication through its seven editor tools. Structural editing required a separate host lifecycle implementation.

The primary comparison runs from the benchmark's timed flow start through publication and verification. Fixture/database preparation and the owner's pre-start workspace/connector handoff are excluded and recorded separately as setup overhead. The editor-only interval and terminal handoff delay are also reported so the timing boundary can be inspected.

Product workflow and guard boundary¶

Workspace makes the import-to-publication workflow available to the editing agent through one MCP connection. An integration still supplies the MCP transport, model session and database, but it needs no separate privileged initialization/export routine for the successful editing path. Independent task verification remains useful in every arm.

Both PatchDIFF arms use explicit observed snapshots, exact scope-local operations, version/hash guards and atomic commits per target. Workspace adds the editor-facing file import and SAVE publication tools. Acceptance establishes a validated state transition. The independent byte oracle and behavioral checks establish correctness for this task. A five-file workflow is not one atomic filesystem transaction; SAVE publishes individual files and reports filesystem and database outcomes separately.

Experimental conditions¶

• Source commit 41ef0e09f06d86f94324800c4c4858074551df69. The packaged adapter and installed adapter were compared with that commit.
• Same original fixture bytes, task and byte oracle across all 18 trials. The task covers Python, JSON and Markdown, repeated similar functions/objects, overlapping matches, selected fields and literal conflict delimiters.
• Antigravity sessions were operated with the requested Gemini 3.8 Flash High selection. Recorded model/reasoning and fresh-session fields are operator attestations. Provider confirmation and exact per-trial Antigravity build identification were not captured. host_build contains the placeholder production.
• Six serial blocks alternate Git and non-Git. Each block contains all three arms. Arm positions rotate within each environment. Each trial has a ten-minute limit.
• PatchDIFF trials use fresh databases in an owned disposable PostgreSQL 16.15 cluster. Resident Waymark 4.1.0 uses Node 22.19.0. Native receives no PatchDIFF tools. Structural exposes five tools and Workspace seven.
• Native uses native file tools. PatchDIFF editors use their advertised catalog, with session help/schema reads allowed. This measures those assigned workflows. It does not measure unrestricted shell, Git CLI or test-running strategies.
• Preparation creates fixture copies, initializes Git where assigned, and creates/migrates a database where assigned. Timed start precedes structural host initialization or Workspace editor import. Prompt submission, editor work, publication, operator terminal handoff and verification occur before the timed end.
• Failures and timeouts would remain in the attempt table. The 18 scheduled attempts all completed successfully. The separate four-trial pilot and unscored smoke checks are excluded.

Trial outcomes and calls¶

Editor calls include auxiliary host tools and completion messages. MCP calls count editing-connection tools/call requests. Host lifecycle calls and file writes are counted separately. Accuracy is checked against all five original paths, including untouched bytes.

All 18 outcomes were success. Independent rechecking found 90/90 byte-exact files, 360/360 behavioral checks, no captured MCP errors/rejections/partial results and no transcript error records. Native receipts report zero operator-classified retries. No comparative accuracy advantage was observed because every arm passed.

Timing, calls and tokens by environment¶

Each environment/arm row contains three trials. Means below use the same three repetitions per cell. The final pooled rows contain six trials per arm. Percentages in the narrative use ratios of arithmetic means.

Workspace’s mean timed interval was 10.6% lower in Git and 21.4% lower in non-Git than Native. The corresponding editor-call reductions were 41.0% and 44.1%. Workspace had a lower timed interval than its same-block Native control in all six blocks. Structural was faster in four of six paired blocks. These are observations from one task and one requested model setting, with three repetitions per cell. No general performance or statistical-significance claim follows from them.

Token totals sum the transcript’s exported input_tokens and cache_read_tokens fields across planner records. They measure cumulative exported token accounting, including repeated cached input. They are not unique context size, billed cost or a direct measurement of reasoning. The editor span runs from the first planner record to the terminal transcript record. Inter-call time mixes model, provider, tool and host scheduling.

Granular tool invocation breakdown¶

The table below details the exact tool invocation frequencies averaged per run across the experimental matrix. In the native baseline, the agent spent the majority of its calls on iterative file reading (read_file) and standard string replacements (edit_file). In structural and workspace modes, Waymark AST structural indexing (list_scopes, observe_scopes) replaced multi-chunk reading with targeted scope observation, while commit_bundle and save handled atomic execution and file write-back.

Setup, publication and verification¶

One-time setup phases totaled 33.946 seconds. The phases were venv creation 9.754, wheel installation 22.229, PostgreSQL package extraction 1.206 and Waymark installation 0.758. This is separate from per-trial preparation and the connector handoff. Runner startup is retained in the packet logs. The runner emits no single bootstrap-duration field.

Preparation and the pre-start owner/connector handoff remain separate from the benchmark's verified interval. Terminal handoff is the interval from the terminal editor record to the operator's finish request. It occurs inside the recorded verified interval and is shown independently rather than attributed to editing or model reasoning. Tool-duration sums can overlap and are not interchangeable with elapsed wall time.

Workspace publication time comes from the last successful SAVE response with publication=installed and durability=confirmed. Captures pair each SAVE with its raw disk digest and the stored checkpoint verification receipt. Structural publication time comes from the last host file-write journal entry after SAVE, successful replay and export. Native publication is observed on disk at finish, which is an upper bound on its last write. The machine-readable evidence retains per-trial start, submission, publication and verification timestamps.

What the captures establish¶

Across six Workspace trials the editor made 30 imports, 30 observations, 30 guarded bundle commits and 30 explicit SAVEs. Across six structural trials it made 30 observations and 30 guarded bundle commits, followed by 120 host lifecycle calls and 30 host file writes. Both PatchDIFF arms completed 20 scope groups per trial. Native made 120 native edit calls across its six trials.

Workspace used 36 scope-list calls and 10 exact-search calls in total. Structural used 36 scope-list calls, 64 searches and two target reads. Schema/help reads and completion messages remain included in editor-call totals. The three-arm design compares complete workflows; it does not independently isolate the effects of tool descriptions, prompt policy, context acquisition and submission batching.

Independent confirmation and reproduction¶

The independent audit re-read the raw transcripts and wire logs, recomputed step/call/token totals and clock intervals, checked all output files against the frozen oracle, reran the 20 checks on each verified snapshot, and reconstructed 60 accepted guarded commits across 240 scope groups through the pinned deterministic batch applier. All reconstructed states match the recorded commit hashes and published file bytes. Structural replay responses and Workspace SAVE/checkpoint verification receipts were cross-checked against the captures.

Both recorded runner teardowns report CLEAN, with owned database clusters returning from 0 → 1 → 0 and no remaining runner lease. The PostgreSQL clusters have been removed, so this audit verifies their captured checkpoint evidence. Capture consistency does not establish provider-side model identity.

The source packet is retained privately as pr24-18trial-production-benchmark. Its raw model transcripts and private host paths are excluded from the repository. The sanitized per-trial evidence pins the source SHA, wheel SHA-256, packet manifest SHA-256 and each transcript SHA-256. With that packet and the tested environment available, reproduce the audit with the installed wheel and evaluator dependencies:

export PATCHDIFF_WAYMARK_MODE=resident
python -B backend/tests/verification/workspace_benchmark_audit.py /absolute/path/to/pr24-18trial-production-benchmark /absolute/path/to/new-audit.json

The audit script enforces that the output path (e.g. /absolute/path/to/new-audit.json) must reside outside the capture packet directory to prevent modifying captured evidence. Running the audit creates its requested output file and checks the existing captures. It performs no model invocation or database restart. Source and test-suite verification remain described in the test matrix.

Interpretation¶

The measured result supports using agent-owned Workspace import, guarded editing and publication for this workload. It combines the per-target transaction guarantees with a complete file-editing interface and reduces the need for host lifecycle orchestration. Timed execution and editor-call totals improved over the assigned Native baseline. Broader tasks, model settings, unattended connector operation, concurrency and semantic safety require their own evidence.