Engine Hardening Log

DistributedEngineeringChangelog

Stable On by default · production-ready · a record, not a feature: everything here is in the repository's commit history and plan documents (violation-registry V-024, cluster-dual-mode-dispatch-plan)

Why this page exists

The engine’s rules say a mechanism that is built, tested by hand-constructing its input, and never wired to the producer that would feed it in production is the recurring defect class. This arc was the closure of that class for distributed execution, and the same rule applied to the arc itself: every phase had to prove itself through its production entry point with a counter that moved. This log is the audit trail.

Before the engine: closing the gap report

The arc opened by working through the distributed-execution gap report prompt by prompt until every closable row was closed. The items that changed code:

• Write-domain container naming — containers are named by table, partition ordinal and epoch window, so a domain’s files identify themselves.
• Vacuum age-out policy — age-based retirement of superseded containers, gated behind AETHERIUS_VACUUM_AGE_OUT=1 after an unconditional version stranded space in the end-to-end test.
• Manifest loader — the cluster manifest loads from disk at boot instead of living only in tests.
• Heartbeat — the standby-aware monitor; a per-tick Vec in its sweep was hoisted out after a commit message overstated its allocation profile.
• Ingest mutex removal — the coarse ingest lock came out. Removing it exposed a same-table writer collision (two writers on one .acf), found in a self-review of the flush physics, and fixed with a per-table exclusive drain (with_table_exclusive) plus a ten-thread chaos test. Mutation-tested: without the lock, or without the drain, zero of 200 base entries survive.
• Segment-cache sharding — attempted, measured five to seven times worse (206–323 → 1,528 waits), reverted, and recorded as a negative result.

The phases

Phase	Landed	Measured
P6 Worker scaffold	AETHERIUS_WORKER_MODE listener, worker-wide fencing epoch, header-only EXEC_SUCCESS	a dummy frame accepted over a socket
P7 Tiny VM	64-bit ISA byte-pinned to the wire struct, VmState with 256 registers, zero-allocation execute_chunk	0.984 ns per instruction
P8 Compiler + wiring	zero-allocation CompilerContext, single-pass expression compiler, full boolean ISA, MockBlock deleted for real .acf stripes, RemoteDispatcher shipping through the FilterCompile seam	25.5 ns per 3-predicate WHERE; 5.88 ns/row over 1 M real rows
Valves lifted	nullable columns through the existing rank dictionary; 4,092 as a chunk size with resume; discovery-driven container iteration; container_id on every chunk (v12)	1.12 ns/instruction nullable; 105,600 matches in 27 chunks
P9 Aggregates, projection, join arena	ACC_* opcodes and one terminal aggregate frame; columnar YIELD_COL chunks; the coordinator join arena made streaming with a zero-allocation remote sink (v13)	COUNT(*) over 10 M rows = 80 bytes; SUM over 5 M rows = 28.5 ms
P10 Loop specialisation, joins, GROUP BY	four loops chosen once per program; HASH_PROBE over a broadcast table (v14); HASH_ACC into 64-byte group slots	dense bench back from 7.3 to 5.78–5.86 ms; first bytecode join, 2,880 pairs
P11 AVG, 1:N, SQL router	AVG as SUM+COUNT; split-array 1:N table (v15) with a per-row match cursor; the two-stage pipeline with measured build-side selection	SQL join across two workers, 30,000 × 2,000 rows
P12 Fat tables, composite keys, post-probe	records carry build-side columns (v16); HASH_MIX + key verification; cross-side conjuncts as bytecode	build-side projection, composite ON, o.amount + c.tier > 1000 from SQL
P13 ALU, LEFT JOIN, Shadow HA	SUB/MUL/DIV (zero → NULL); HASH_PROBE_OUTER; the shadow streamer wired over TCP with header refresh (v17)	LEFT JOIN from SQL; two deltas byte-identical with a live-appending primary

Bugs the tests caught, in the order they were caught

1. A compiler edit that silently did not apply left every new operator returning Unsupported — caught by the operator oracle test; edits are now asserted.
2. A vacuous mutation (an AND jump one short still passed) — replaced with jump-past-epilogue and drop-short-circuit mutations, both red.
3. The same-table writer collision after the ingest mutex was removed.
4. SUM of zero rows returned 0 instead of NULL — accumulators now start NULL.
5. A refactor emitted YIELD_MATCH twice; every projected row came back doubled.
6. A NULL embedded build cell read as valid when the probe column was dense — the JOIN loop now always tracks NULL.
7. Process-global counters read by two tests in one binary — asserts became “moved”, not “by one”; once-per-process installs (topology, compiler hook) got their own binaries.
8. A pure tail stream left the shadow’s container header stale after the second delta — the header region is re-shipped after every delta.
9. Two docs commits claimed a plan paragraph that had not landed (the anchor wrapped mid-phrase) — inserted by line, with a correcting message.

The regalloc investigation

Adding aggregate and projection arms to the single VM loop moved the dense 1 M filter bench from 5.90 to 7.3 ms. Four structurally equivalent shapes measured 5.8–8.2 ms with run-to-run spread under 1 percent and no consistent direction. The alignment hypothesis was tested with an aligned-loops build (-C llvm-args=-align-loops=64) in a separate target directory: it reproduced the numbers to 0.1 ms, so alignment was falsified. The cause is register allocation across one large match function. The fix was one loop per program kind, and the dense bench is re-run after every change to the loop.

The discipline

• Counters, not clocks. Every fast path exposes a counter and the test asserts it moved. Wall time is never the evidence a branch fired.
• Mutation-test the guard. Break it, watch the test go red, restore. The zero-allocation guard proved itself by going red on a stray allocation during development.
• Never diff n = 1. Benchmarks are re-run; a build is compared to a baseline measured the same way.
• 200-line files. Every source file under crates/ stays under 200 lines; the gate ran before every commit and forced the VM’s loop split, the worker’s module layout, and the compiler’s helper modules.
• Logical commits. Each phase landed as three to five commits — wire, runtime, orchestrator, docs — each with its red-flag answers (allocations per row, recovery, indexes) in the message.

After the arc: tenants, transactions, indexes, strings

The same discipline carried the next five phases, each on its own branch:

• Fluid Clusters — every frame names its tenant; a tenant’s files live in their own directory; the compiler reads virtual and per-tenant columns from a schema the application installs. Wire v18–v19.

• Tenant operations — NULL defaults, a per-tenant in-flight quota that refuses before a worker is touched (measured: exactly the configured number of rejections under a hundred racing statements), and hardlink snapshots with a restore that is a routing change.

• Vacuum and the security gate — the multi-tenancy flag that both ends enforce; DELETE as tombstones; a background compactor whose file switch is one pointer store, proven under a SELECT running through the pivot; then UPDATE as delete-plus-insert. Wire v20–v21.

• Distributed MVCC — a 64-wide commit window every frame carries, so a worker decides visibility with no round trip; rollback is one bit; write conflicts answer 40001 instead of waiting; a watchdog reaps stalled transactions; aborted rows and files are purged by the vacuum; a restarted orchestrator rebuilds its numbering from the workers. Wire v22–v25.

• Secondary indexes, VARCHAR, and string ranges — a sorted index sidecar built on seal; point lookups, then ranges and LIKE 'prefix%'; text columns with short strings inline and long ones in a per-file store; a text index keyed on eight-byte prefixes with every hit verified against the whole string. The range integration test caught a COUNT that summed 1..100 instead of counting 100, and the release bench caught a 4.7 % regression from an inlined string compare — both fixed before they shipped. Wire v26–v27.

• The first real two-process boot (2026-09-05) — an orchestrator daemon, a worker daemon and psql, which no in-process test had ever assembled — could not serve one query: the worker’s heartbeat reply carried a fresh timestamp and a container id where the monitor expected an echo, so every reply was a strike and a healthy worker was declared dead. Fixed at wire v30 with a regression test that drives the real monitor probe at a real worker. In the same pass: snapshots had linked only the .acf (a restore would have resurrected deleted rows), the vacuum would have compacted an immutable snapshot, and the radix-shuffle receiver lived per connection so peers on separate sockets could never complete a join. All four were found by tests written against the production shape rather than a hand-built input.

Wire versions in this arc

v10 → v17 for the execution engine, v18 → v31 for the phases above. Each bump and its reason is in Wire Protocol Versions.

Cluster Execution Engine →

The result of the arc, as a system.

Deterministic Simulation Testing →

The engine's wider testing discipline.

Observability & Error Taxonomy →

Where the counters surface.