PatchDIFF Product Change Record¶

PatchDIFF runs three declarative transaction formats through one applier and locked PostgreSQL submission path. Hosts can read a target, preview a proposal, recover a recorded outcome and verify history through replay. PostgreSQL owns the working target.

Architecture and contracts: architecture, MCP tools, core runtime, storage, transaction formats and host guide.

Workspace editing and publication¶

The five-tool workspace and seven-tool observed_workspace profiles start with an eager validation of the project directory lease and database migration revision. Initialization imports a file using a relative path within the workspace root (from_workspace: true) and returns the target ID without returning source text. The editing agent submits guarded text edits through workspace (patchdiff_submit_edits) or guarded structural scope bundles through observed_workspace (patchdiff_commit_bundle) to update PostgreSQL as the authoritative working copy.

Workspace publication is executed explicitly via patchdiff_save with sync_workspace: true: - State Guards & Directory Verification: Verifies expected_target_version and/or expected_current_hash under SELECT ... FOR UPDATE, and confirms active directory lease integrity. - Disk Drift Detection: Reads the physical file bytes currently on disk and computes their SHA-256 digest, comparing against the last observed digest. If external modifications have occurred, the save operation aborts with a drift rejection, leaving both disk and database unmodified. - Canonical Installation: Converts canonical LF content using the target's recorded source_line_endings and writes the exact bytes to disk. - Partial SAVE Recovery Boundary: The physical file write and the database checkpoint commit operate across distinct failure domains. If file installation succeeds but the database checkpoint transaction fails, patchdiff_save returns status: "partial" with publication evidence and an unconfirmed database checkpoint receipt, enabling explicit reconciliation without data loss.

Two verified installation and execution pathways are supported: 1. Pure Python 5-tool workspace: Starts without Node.js or external toolchains, exposing exact text editing (initialize_target, read_target, find_occurrences, submit_edits, save). 2. 7-tool observed_workspace: Uses the Waymark bridge packaged directly with the wheel, requiring Node 22+ and npm setup, enabling structural AST scope discovery, observation, and bundle commits (list_scopes, observe_scopes, commit_bundle) alongside initialize_target and save.

Installed-wheel checks used PostgreSQL 16 and live stdio to verify the working copy before SAVE and exact file bytes and checkpoint afterward. The PR #24 benchmark provides an authoritative 18-trial comparison of Native, host-orchestrated structural editing, and agent-owned Workspace editing across 90 files and 360 behavioral checks (all passed). Workspace eliminated host orchestration routines and completed the workflow in 16.2% less time with 42.7% fewer editor calls. Preparation and owner handoffs are recorded as separate setup overhead. Model/build attribution and the timing boundary are recorded in that report.

Transaction formats and retirements¶

Raw text-delimited patch syntax submission (patchdiff_submit), the raw text parser, and prompt-context regeneration tools (patchdiff_get_regeneration_context) have been purged entirely from the architecture. All mutation is natively typed JSON (patchdiff-tx/1, patchdiff-tx/2, patchdiff-tx/3, and patchdiff-observed-batch/1). The dry-run tool patchdiff_preview supports the three typed formats (tx/1, tx/2, tx/3), projecting execution facts within a 32,768-byte budget without writing database rows or reserving locks.

External PATCHDIFF_MCP_RESPONSE_MODE selects immutable detailed (default) or compact presentation. Only observed acceptance, candidate discovery, and bundled observation successes are projected. Full audit facts, all warnings, guards, identities, exact source, and continuations remain available as contracted; errors and unrelated responses are unchanged. A separate trusted scalar finalization recipe checkpoints, replays, and exports accepted target states without model lifecycle turns, with explicit SAVE uncertainty and partial-failure reporting. See MCP tool contracts and Host Integration Guide.

Structural batches extend observation tools to support atomic multi-scope editing without adding redundant tool names. In PR24, the Waymark coordinate integration was updated with a half-open coordinate range fix, ensuring that AST symbol boundaries align precisely with half-open [start, end) Unicode code-point spans used by the core applier. In the primary observed_workspace profile, five structural routes (list_scopes, observe_scopes, commit_bundle, find_occurrences, read_target) are exposed alongside agent-owned initialize_target and save. patchdiff_observe_scopes issues 1 to 32 disjoint scopes from one snapshot; patchdiff_commit_bundle requires both guards and a request key, evaluates scope-local tx/2 operations through the shared applier, and commits one target increment. The internal durable pair is patchdiff-observed-batch/1 / patchdiff-observed-batch-r1; historical single-scope records retain their semantics. Migration 0005 added bundles and many-to-one accepted linkage, and replay verifies every recorded scope without Waymark. See Structural Batch Editing.

PR24 implements 10 startup profiles in code (editor default 7, unobserved 5, observed 6, observed_editor 4, unobserved_editor 4, hybrid_editor 5, workspace 5, observed_workspace 7, observed_v2 5, and explicit trusted lifecycle full 17). External-facing agent documentation focuses on the three active operational profiles corresponding to the benchmark arms: 1. observed_workspace (Primary): 7 tools (initialize_target, read_target, find_occurrences, list_scopes, observe_scopes, commit_bundle, save). 2. workspace: 5 tools (initialize_target, read_target, find_occurrences, submit_edits, save). 3. observed_v2: 5 tools (read_target, find_occurrences, list_scopes, observe_scopes, commit_bundle).

Profiles filter both tool discovery (tools/list) and call dispatch, remaining immutable for a session. The internal service registry implements all 17 underlying tool handlers. Trusted clients that need export, administrative history, or replay select full before starting their session.

Fresh replacement raw text and canonical result each have inclusive limits of 1,000,000 code points and 1,048,576 UTF-8 bytes. Invalid Unicode follows the audited encoding-rejection path. Existing-key recovery/conflict precedes fresh admission. Oversized valid fresh replacements return invalid_argument/replacement_too_large without writes; replay does not reapply fresh admission limits. Runtime 0.3.0 continues to reject replace_all.

Profiles are immutable. Replay dispatches on the stored protocol/runtime pair, preserving 0.1.0 non-overlapping matching, 0.2.0 historical typed change-ID grammar, 0.2.1 strict typed IDs, and the pre-0.2.2 ignored-singleton-selector behavior. Current match-capable profiles enforce every supplied selector, even for one occurrence. Runtime 0.5.0 adds only span context guards; tx/2 records without them keep 0.3.0. Unknown protocol/profile pairs fail closed; historical records and fingerprints are not rewritten.

Atomicity, identity and audit¶

Each accepted transaction applies all changes sequentially in memory and commits atomically per target under SELECT ... FOR UPDATE. The append-only attempt log records admitted accepted and rejected submissions; the accepted-only ledger carries origin and runtime-derived spans, character counts and hashes. ValidationStage determines validity; task correctness and semantics remain not_evaluated.

Exact keyed recovery returns the original outcome before stale guards and appends no row. Conflicting reuse creates a distinct rejected NULL-key row with the submitted fingerprint, original-log linkage and bounded key preview. The original keyed row stays immutable. Request identity includes the full canonical raw proposal and guards; sorted JSON keys do not normalize decoded strings, omitted/null edit fields, advisory prose or selectors. Eligible same-target rejected parent UUIDs canonicalize; ineligible parent spellings retain their existing rejection identity.

Unpaired surrogates and NUL are rejected without repair. Known-target invalid input is preserved reversibly as ASCII-escaped patchdiff-invalid-input/1 evidence in existing TEXT fields, with safe JSONB diagnostics. Valid serialization, hashes and historical recovery identities stay unchanged. Read/search text argument errors create no edit-attempt row. Stdio explicitly uses strict UTF-8 input/output, including on Windows; malformed wire bytes are not repaired.

Locked submission and replay reads refresh canonical state. PostgreSQL READ COMMITTED supports lock-then-read recovery; callers selecting stronger isolation own serialization/deadlock retries. Caller-owned transactions retain commit/rollback and lock-lifetime ownership; infrastructure errors are not masked as validation rejections.

Bounded reads, diagnostics and preview¶

Offset reads default to 4,000 and cap at 8,000 code points. Line reads use one coherent ordinary column observation under no_autoflush, inclusive one-based LF-only ranges, trailing-newline inclusion and the final empty line. Continuation is relative to the selected range (next_offset) or absolute (next_start), and opt-in line_starts give absolute line offsets; no local state is flushed. Exact search counts overlapping occurrences and caps context at 400 code points. History defaults to 50 rows, caps at 200 and uses exclusive sequence cursors; attempts descend, while each selected newest ledger page displays ascending.

Rejections preserve immutable original diagnostic.failure_context and expose separate fresh response-only current_context. A known failure location beyond the start of the text makes failure_context a window around it; an MCP current_context identical to it is returned as unavailable (same_as_failure_context). Each has at most 256 code points and 4,096 compact ASCII-escaped JSON bytes; together they fit 8,192 bytes including wrappers. Canonical/intermediate buffer labels and prior-change counts do not imply a partial commit. Older diagnostics are never retrofitted. Optional correlation is a fresh invocation UUID, including recovery, and never changes durable identity.

Exact diagnostic candidates are limited to five, with 20-code-point side contexts, a 64-code-point match preview and an 8,192-byte wrapper. Immutable failed-query evidence is capped at 64 code points and 1,024 wrapped bytes. Near misses are response-only after successful service-owned commit/lock release, explicitly omitted for caller-owned transactions and for consumed sequential matches. Fresh rejections scan a never-persisted copy of up to 65,536 code points of the failing buffer; exact recovery scans the stored context. Distances use an exact Ukkonen-banded Landau-Vishkin Levenshtein program (band_radius = (threshold + 1) // 2) with exact positional pigeonhole filtering within a 1,000,000-cell cap; reaching the cap stops at a window-start boundary and reports the scanned range. Windows are query length ±1, the threshold is min(4,max(1,length//4)), and up to three non-overlapping candidates are ranked by distance/start/end, with 16-code-point side contexts and an 8,192-byte wrapper. Combined new enrichment is capped at 18,432 compact ASCII-escaped JSON bytes, excluding preexisting arbitrary evidence. Suggestions never select or apply a fuzzy edit. Span failures report span_check, buffer_length and up to 64 code points of actual_text; sequential_match_invalid names the earlier change that consumed each original occurrence. diagnostic.retryable follows one shared definition.

Accepted responses carry current_version/current_hash next to version/after_hash, plus advisory warnings (no_op_change, no_op_transaction, occurrence_index_basis_shift, grapheme_boundary_split) that are stored on accepted rows only when present and returned on exact recovery. Request-key conflicts carry diagnostic.code: "request_key_conflict" with the differing fingerprint components and record fields. Tool-result text is literal inside ASCII-escaped JSON-RPC lines, and unknown or missing tool arguments return a structured invalid_argument payload.

Preview uses the shared fresh guard/decode/apply evaluator, one coherent observation and an explicit format. It has no request key, lineage, audit, writes, locks, commit or rollback; pending local target changes are rejected without flushing. Rejected previews return no partial facts. Submission always re-locks and rechecks; a preview reserves no state.

The complete preview payload is capped at 32,768 compact ASCII-escaped JSON bytes, excluding transport envelopes. Accepted facts form a prefix of at most 32 complete facts, 2,048 bytes each and 16,384 bytes including the wrapper. Text evidence caps at 1,024 code points/8,192 wrapped bytes per field; diagnostic wrappers cap at 16,384 bytes. Explicit counts and omission metadata retain verdict, reached stage, valid target/observed identity and accepted hash. Oversized identities are omitted, never shortened. MCP tool contracts defines deterministic omission order. Ordinary submission/recovery facts and stored audit evidence remain complete.

Discovery examples are deep-copied per result and bounded: 65,536 compact ASCII-escaped JSON bytes for discovery, 8,192 bytes for the complete examples map. The host guide gives the executable sequence binding returned IDs, guards, parents and cursors, including deduplication and complete history traversal.

SAVE, replay and schema¶

SAVE snapshots current target state and rotates editing cycles without mutating content; optional version and hash guards are both checked. Replay checks initial snapshot ownership/hash, accepted-log ownership/content hashes, contiguous versions, before/after hashes and recorded execution profiles. Materialized health reports independent raw content_matches, stored_hash_matches, recomputed_hash_matches and version_matches; successful reconstruction can coexist with unhealthy storage. CRLF/bare-CR corruption is visible even if canonical hashes agree. Replay never repairs target or history.

Migrations are 20260510_0001, 20261020_0002, 20261022_0003, 20261026_0004 (observations), 20261027_0005 (bundles) and 20261105_0006 (typed-origin adoption). Migration 0002 adds ledger origin/derived facts, request key/fingerprint and the target/key partial unique index; typed ledger prose may be null. Downgrade refuses when it would discard typed audit evidence. Migration 0003 adds nullable cloud_target_files.source_line_endings, recorded for new uploads before LF canonicalization; existing targets keep NULL, and downgrade refuses while recorded facts exist. patchdiff_export_target honours original_text (canonical LF, the default), canonical_lf, crlf and original_eol and refuses other formats. Ledger and attempt pages expose their join keys, bounded intent and a per-page content byte budget without schema changes.

Additive migration 20261105_0006_typed_origin.py applies the typed ledger default and check constraint (ck_append_only_manifest_ledger_origin) to existing as well as fresh databases under an exclusive table lock. It inspects all existing rows in append_only_manifest_ledger and explicitly raises an exception if any non-typed origin is present, refusing upgrade without modifying audit history. It sets server_default = text("'typed'") and enforces origin = 'typed' without rewriting historical audit records. Historical 0002 retains its original default and protects origin/derived facts and request identity on downgrade. See the data-model upgrade contract.

Verification scope¶

The Test Matrix maps unit, live PostgreSQL, populated migration preservation/refusal, and real stdio workflows. The automated verification suite comprises 1,429 automated tests: 923 unit tests (with 1 Windows-only skip for socket transports) and 506 live PostgreSQL integration tests, plus migration convergence across all six schema revisions. Verification documentation provides reproduction commands. Recorded deterministic byte accounting (151 B wire for 1 minimal tx/1 edit, 259 B for 5 edits, 341 B tools/call line for small tx/3 replacement), SVC-001 65,536-code-point near-miss latency (1.7 ms on varied prose, 41.9 ms with 100% scan coverage on repetitive logs), and MCP-001 PostgreSQL 16+ LATERAL history/ledger latency over 20 MB substrates (14.68 ms ledger / 22.85 ms log on 50 × 400 KB tx/3; 136.72 ms log / 221.59 ms ledger on 20 × 1 MB tx/1). Test counts and CI results apply only to their executed revision; this record does not certify current publication CI.

Live-model performance is established separately by the Workspace E2E Benchmark, an empirical evaluation across 18 full trials (90/90 byte-exact files, 360/360 behavioral checks passed) comparing Native, host-orchestrated structural editing, and agent-owned Workspace editing. Deterministic offline oracles (payload byte accounting, near-miss DP benchmarks, and SQL LATERAL pagination benchmarks) measure algorithmic and storage properties and remain strictly separated from live-model claims.

Historical source and verification records remain preserved in Git and internal archives; they do not establish verification of a later revision.

Historical PR20: opt-in purpose-specific editor contract¶

Decision: add observed_v2 with eight external purposes, six common bundle paths and two single-scope compatibility paths. The latter preserve scopes above the batch issuance cap. Reusable explicit snapshots remove result-to-argument field renaming; bounded receipts preserve immutable transaction evidence separately from current state. Modern MCP revisions are negotiated with text fallback and optional structured results/output schemas. Existing profiles, 22 internal handlers, services, transaction formats, migrations and replay runtimes retain their contracts.

The full contract records compatibility, audited admission/recovery ordering, response limits, host-loading uncertainty and actual verification paths. This is a tool/schema/data change; the rejected prompt-only proposal is not implemented. Static bytes and backend correctness do not establish model-time savings.

Historical PR21: streamlined five-tool observed editor surface & receipt compaction¶

Decision: streamline observed_v2 to five external purpose-specific tools (patchdiff_read_target, patchdiff_find_occurrences, patchdiff_list_scopes, patchdiff_observe_scopes, and patchdiff_commit_bundle), eliminating dual-path confusion by pruning redundant singular tools (patchdiff_observe_one, patchdiff_commit_observation, and patchdiff_read_observation). Resolve commit receipt bloat and client response truncation by preserving the authoritative scopes confirmation array while dropping raw changes span facts (reducing receipt payload size from ~12 KB to ~2 KB). Retain rich candidate symbol and selector metadata in patchdiff_list_scopes for non-blind scope discovery.

The PR 21 retrospective records the full architectural narrative across PR 19, 20, 20slim, and 21, and documents the empirical 5-file canary benchmark (24 planner steps vs 41 in PR 20, 16 MCP calls vs 35, 968K cache tokens vs 2.64M, 0 rejections/retries, and 100% full-byte correctness). Existing profiles, internal handlers, services, transaction formats, migrations, and replay runtimes retain their contracts.