Wire Protocol Versions
DistributedReferenceWire Protocol
Beta Works · surface still evolving · a peer on a different version refuses the frame at the header; every layout below is pinned by const assertions and a layout test in the wire crate
The rule that governs every change
Section titled “The rule that governs every change”A new discriminant, a re-carved field, or a reshaped payload under an unchanged version is the exact shape of the bug that once booted a database empty and reported success: an older peer reads the bytes plausibly and wrong. So the version is bumped for every such change, and the layout test asserts the current number with the reason in its message. Peers compare versions in the header before anything else.
The header (64 bytes, #[repr(C, align(8))])
Section titled “The header (64 bytes, #[repr(C, align(8))])”| Offset | Field | Notes |
|---|---|---|
| 0 | magic: [u8; 4] | AEWR |
| 4 | version: u16 | 17 |
| 6 | message_type: u16 | raw discriminant; decoded, never transmuted |
| 8 | target_cluster_hash: u64 | cluster_hash(name) — a format-stable hash both processes compute with the same code |
| 16 | instruction_count: u32 | 64-bit words following the header |
| 20 | constants_len: u32 | payload bytes following the instructions |
| 24 | header_checksum: u32 | FNV-1a over the header; setters re-seal |
| 28 | _pad: u32 | zero |
| 32 | manifest_epoch: u64 | fencing token; a worker refuses an epoch older than it has seen |
| 40 | txn_id: u64 | transaction frames only (v10) |
| 48 | container_id: u64 | which container a result chunk indexes (v12) |
| 56 | reserved: [u8; 8] | zero |
The frame’s total length is 64 + instruction_count × 8 + constants_len.
No byte is compiler padding, so the header is cast to and from bytes
directly.
Message types
Section titled “Message types”| Code | Name | Direction | Since |
|---|---|---|---|
| 0x01 | EXEC_BYTECODE_REQ | orchestrator → worker | — |
| 0x02 | RESULT_STREAM_CHUNK | worker → orchestrator | — |
| 0x04 | EXEC_ABORT | worker → orchestrator | — |
| 0x05 | FETCH_PROJECTION_REQ | orchestrator → worker | — |
| 0x06 | FETCH_STREAM_CHUNK | worker → orchestrator | — |
| 0x07 / 0x08 | HEARTBEAT_PING / HEARTBEAT_PONG | both | v3 |
| 0x09–0x0C, 0x10 | MIGRATE_* | orchestrator → worker | v3 |
| 0x0C–0x0F | VALIDATE_TXN_*, COMMIT_TXN, ABORT_TXN | both | — |
| 0x11 | WORKER_READY | worker → orchestrator | v6 |
| 0x12 / 0x13 | CONTAINER_RETIRED / TOPOLOGY_DELTA | orchestrator | — |
| 0x14 | EXEC_SUCCESS | worker → orchestrator | v11 |
| 0x15 | RESULT_AGGREGATE_CHUNK (terminal) | worker → orchestrator | v13 |
| 0x16 | RESULT_MATERIALIZE_CHUNK | worker → orchestrator | v13 |
| 0x17 | JOIN_STREAM_CHUNK | orchestrator → next stage | v13 |
| 0x18 | HASH_TABLE_LOAD | orchestrator → worker | v14 (payload reshaped v15, v16) |
| 0x19 | RESULT_GROUPED_AGG_CHUNK | worker → orchestrator | v14 |
| 0x1A | SEGMENT_FETCH_REQ | orchestrator → primary | v17 |
| 0x1B | SEGMENT_STREAM_CHUNK | primary → orchestrator → shadow | v17 |
| 0x1C | SNAPSHOT_REQ | orchestrator → worker | v19 |
| 0x1D | EXEC_DELETE_REQ | orchestrator → worker | v20 |
| 0x1E | EXEC_UPDATE_READ_REQ | orchestrator → worker | v21 |
| 0x1F | WRITE_CONFLICT | worker → orchestrator | v24 |
| 0x20 / 0x21 | SYNC_TXID_REQ / SYNC_TXID_RESP | both | v25 |
| 0x22 | INDEX_BUILD_REQ | orchestrator → worker | v26 |
| 0x23 | STRING_ARENA_CHUNK | orchestrator → worker | v27 |
| 0x24 / 0x25 | SHUFFLE_DATA_REQ / SHUFFLE_EOF | worker → worker (via the orchestrator’s router) | v29 |
| 0x26 | JOIN_BROADCAST_REQ | orchestrator → worker | v31 |
Payload layouts
Section titled “Payload layouts”All records are #[repr(C)] with 8-byte fields and no padding, so a payload
is a slice cast, checked for alignment and exact length, never a parse.
| Frame | Payload |
|---|---|
EXEC_BYTECODE_REQ | instruction_count 64-bit instructions, then the constants pool; the first 8 bytes of the pool are the target table id. |
RESULT_STREAM_CHUNK | KeyOffset { sort_key: u64, row_offset: u64 } × n, at most 4,092 (fills the 64 KiB ring). |
RESULT_MATERIALIZE_CHUNK | slot_count: u32, row_count: u32; slot-major values (row_count × 8 B per slot); per-slot validity bitmaps (LSB-first, padded to 8 B); optionally row_count row offsets. |
RESULT_AGGREGATE_CHUNK | { value: i64, valid: u64 } × accumulators, program order. Terminal: no EXEC_SUCCESS follows. |
RESULT_GROUPED_AGG_CHUNK | GroupSlot × n (64 B each: key, used, acc[4], valid[4], pad), at most 1,023. |
HASH_TABLE_LOAD | dir_slots: u64, record_count: u64, stride: u64; directory of KeyOffset { composite_key, start << 32 | count } (power of two, row_offset == u64::MAX free); records of stride words: [row_offset, valid_mask, cols…]. |
JOIN_STREAM_CHUNK | MatchedPair { offset_a: u64, offset_b: u64 } × n; constants_len = 16 × n. |
HEARTBEAT_PING/PONG | Heartbeat { timestamp_us, worker_id, data_end_offset } (24 B). A worker’s PONG carries its first container’s committed end and id. |
SEGMENT_FETCH_REQ | from: u64, len: u64; the container in the header (0 = first). |
SEGMENT_STREAM_CHUNK | from: u64, then the bytes; container in the header. |
Version history (this arc)
Section titled “Version history (this arc)”| Version | Change | Why a bump |
|---|---|---|
| v10 | txn_id carved from reserved at offset 40 | transaction frames used to overload the routing hash |
| v11 | EXEC_SUCCESS | new discriminant |
| v12 | container_id at offset 48; reserved 16 → 8 | a worker walks every container; an offset needs its container |
| v13 | aggregate, materialize and join-stream chunks | three new discriminants |
| v14 | HASH_TABLE_LOAD, RESULT_GROUPED_AGG_CHUNK | two new discriminants |
| v15 | join table becomes 1:N (directory + matches) | a v14 peer would read the preamble as two slots and every probe would miss |
| v16 | join records become fat ([offset, valid_mask, cols…], stride in the preamble); materialize chunks may carry a trailing offset column | payload reshaped |
| v17 | SEGMENT_FETCH_REQ, SEGMENT_STREAM_CHUNK | the shadow tail stream |
| v18 | tenant_id at header offset 56 | every frame names its tenant; 0 is the system tenant, which is what an older peer writes there |
| v19 | LOAD_NULL; SNAPSHOT_REQ; a snapshot word in exec and segment frames | virtual NULL columns; tenant snapshots and restore routing |
| v20 | EXEC_DELETE_REQ | new discriminant |
| v21 | EXEC_UPDATE_READ_REQ | new discriminant |
| v22 | transaction window words in the exec constants | a worker decides row visibility from the frame alone |
| v23 | abort_mask word | a reaped transaction stays invisible after the window moves past it |
| v24 | WRITE_CONFLICT | new discriminant |
| v25 | SYNC_TXID_REQ / SYNC_TXID_RESP | orchestrator boot recovery |
| v26 | INDEX_BUILD_REQ | new discriminant |
| v27 | STRING_ARENA_CHUNK; a string-literal pool after the fixed constants | VARCHAR: an older worker would ignore the pool and answer every string compare false, so this is a refusal instead |
| v28 | unassigned | — |
| v29 | SHUFFLE_DATA_REQ / SHUFFLE_EOF | radix hash-shuffle joins between workers |
| v30 | the heartbeat reply echoes the ping; container id in the header | before this a healthy worker was scored dead and every query refused — found by the first real two-process boot |
| v31 | JOIN_BROADCAST_REQ | a flat join arena as an alternative broadcast build side |
One hash, one crate
Section titled “One hash, one crate”splitmix64, mix and composite_key live in the wire crate so that the
orchestrator building a join table and the worker probing it link the
same code — the same reason cluster_hash lives there. A per-process
seeded hasher would make every probe miss and read as a join bug rather
than a hashing one.