Shadow HA Streamer

DistributedHigh AvailabilityReplication

Beta Works · surface still evolving · wired end to end and proven byte-identical across two deltas with a live-appending primary; one container per shadow cluster in this slice

★

Premium feature Available only with a license key that grants replication. Flags and environment variables cannot enable it. To license this feature, contact the AetheriusDB team at aetheriusdb.com or aetheriuslabs.com. Covers standby VMs and the shadow stream; the daemon carries and reports this grant today and gates the standby topology on it in an upcoming release. See Licensing → Premium features.

What it is

A cluster whose topology entry declares a standby gets a Shadow stream: the standby holds a byte-for-byte copy of the primary’s .acf containers, kept current asynchronously. The mechanism — sync state, the ship loop, the segment streamer — had existed in the wire crate and been tested against in-memory links; this arc gave it real sockets, real workers, and a boot hook.

                 PING every 100 ms            SEGMENT_FETCH_REQ [from, len]
 orchestrator ──────────────────▶ primary ◀──────────────────────────┐
     │  ◀── PONG {data_end_offset, container id}                     │
     │                                                               │ positional read,
     │ observe_primary(offset) → state                               │ fresh descriptor
     │                                                               │
     │  ship [shipped, primary)  ────────── SEGMENT_STREAM_CHUNK ────▶ shadow (append)
     │  then the 8 KiB header region ────── SEGMENT_STREAM_CHUNK ────▶ shadow (overwrite)

How it runs

Two threads per Shadow cluster (background work in this crate is OS threads; no async runtime is a dependency):

1. The poller PINGs the primary on a dedicated socket and feeds each PONG’s committed data_end_offset into the ShadowSyncState. The PONG also names the container (in worker_id), so the shadow’s copy is named identically.
2. The ship loop (spawn_shadow_sync, unchanged) wakes when the primary’s offset is ahead of what has shipped and streams [shipped, primary) through a ShadowLink. The link’s read side asks the primary for at most 60 KiB per SEGMENT_FETCH_REQ; its write side sends each chunk to the shadow as a SEGMENT_STREAM_CHUNK and waits for the ack.
3. After every delta the header region is re-shipped. A container’s header lives at the front of the file and is rewritten in place on every commit; a pure tail stream leaves the shadow’s header stale. Data first, then the header that points at it — the primary’s own commit order. This was found by the end-to-end test: bytes matched after the first delta and diverged after the second.

The non-blocking guarantee

The primary serves a fetch with read_exact_at on a descriptor it opens for that request. It never touches its append writer, holds no lock, and is only ever asked for bytes below a data_end_offset it has already committed. The test appends 5,000 rows to the primary while the streamer runs; the append completes in milliseconds and the streamer records zero stalls.

What the test proves

Step	Result
20,000 rows tail to an empty standby	shadow bytes == primary bytes up to the committed end
Primary appends 5,000 rows during streaming	append not blocked; second delta and refreshed header arrive
Bytes compared again	identical up to the new committed end; zero stalls
A worker boots on the shadow copy	COUNT(*) = 25,000, the same as a fresh worker on the primary

The original primary worker still answers 20,000: workers discover their containers at boot, and re-discovery on append is a recorded follow-up. The streamer replicates bytes; the bytes are what the test proves.

Health and observability

ShadowSyncState exposes the primary offset, the shipped offset, lag in bytes, bytes shipped and stall count; ShadowHealth::Lagging is logged when the standby falls more than 512 MiB behind and is then “not a credible promotion target”. The poller’s pongs_observed counter is the witness that the offset is observed rather than assumed; the worker counts segments_served and segments_appended.

Boot

topology_boot::init spawns one stream for every cluster in the topology file that declares a standby, right after the topology is installed, and keeps the handles for the life of the process. The standby is the shadow; the manifest’s ReplicationStrategy is not consulted yet.

Scope, recorded

• One container per shadow cluster — the PONG names the first. Multi-container tails are the next slice.
• The header refresh assumes the v3 dual-slot layout (8 KiB).
• Workers discover containers at boot only.
• Promotion of a shadow to primary is the existing promote_shadow path; this page is about keeping it current.

Worker Node →

The PING, fetch and append handlers on the worker.

Wire Protocol Versions →

The v17 segment frames.

Crash Recovery & Durability →

Why a container's header is rewritten in place on commit.