Worker Node

DistributedStorageBytecode VM

Beta Works · surface still evolving · boots from AETHERIUS_WORKER_MODE; serves real .acf containers; proven over loopback TCP with two-worker joins and a live-appending primary

★

Premium feature Available only with a license key that grants cluster. Flags and environment variables cannot enable it. To license this feature, contact the AetheriusDB team at aetheriusdb.com or aetheriuslabs.com. A worker started without the grant exits instead of serving. See Licensing → Premium features.

What it is

A worker is the daemon booted with AETHERIUS_WORKER_MODE=<cluster>. It binds one TCP port and nothing else — no pgwire, no native SQL, no catalog. Its whole job is to take a frame of instructions and run it over the column stripes it owns.

AETHERIUS_WORKER_MODE=orders_domain \
AETHERIUS_WORKER_PORT=5434 \
AETHERIUS_WORKER_DATA_DIR=/var/lib/aetherius/containers \
aetheriusd

Boot: discovery

The worker learns its data from its own disk, not from the orchestrator. At boot it scans the containers/ directory with the container map, opens every row-bearing entry (Base and DeltaInsert) of every table as an AcubeBlock, and keeps the list immutable for the life of the process. One unreadable container is skipped and counted, not fatal.

Opening a block is the only cold work: one directory decode and, for each nullable column, one rank-dictionary build. From then on a program’s LOAD_COLs are resolved to borrowed slices before the row loop starts, so the loop does no lookup, no decode and no allocation.

A worker discovers containers once, at boot. A container appended after boot is served by the next worker that boots on the directory, not by the running one. Re-discovery is recorded as a follow-up.

The connection

Each accepted socket gets its own thread, a 1 MiB inbound buffer, and one Scratch: the chunk buffers for matches, reply bytes, projected cells and validity, a 65,536-slot broadcast join table with its records, and a 4,097-slot group table. That is the only allocation the connection ever makes. Frames are then handled without touching the heap:

Frame	What the worker does
EXEC_BYTECODE_REQ	Reads the target table id from the constants pool, runs the program over every block of that table, streams the result shape the program produces, then EXEC_SUCCESS (or the terminal aggregate frame).
HASH_TABLE_LOAD	Copies the broadcast join table (directory, fat records, stride) into the connection’s scratch and acks EXEC_SUCCESS.
HEARTBEAT_PING	Replies HEARTBEAT_PONG carrying the committed data_end_offset of its first container and that container’s id.
SEGMENT_FETCH_REQ	Positional read (read_exact_at) of the requested committed byte range on a fresh descriptor; replies one SEGMENT_STREAM_CHUNK.
SEGMENT_STREAM_CHUNK	As a shadow: writes the bytes into its copy at the stated offset — an append at the current length, or an overwrite inside the 8 KiB header region — and acks.
A frame behind the worker’s fencing epoch	EXEC_ABORT; counted as stale.

A frame from a stale orchestrator (an older manifest epoch than the worker has already seen) is refused before any of this runs — the worker-wide FencingEpoch is what stops a superseded coordinator from driving writes.

Result shapes

The program’s shape (vm::Shape) picks the reply:

Shape	Frames	Chunk bound
Filter	RESULT_STREAM_CHUNKs of KeyOffset { sort_key, row_offset } + EXEC_SUCCESS	4,092 offsets
Materialize / Join	RESULT_MATERIALIZE_CHUNKs: slot_count, row_count, slot-major values, per-slot validity bitmaps, optional trailing offsets + EXEC_SUCCESS	derived from the 64 KiB ring for the slot count
Aggregate	one RESULT_AGGREGATE_CHUNK of {value, valid} records, terminal	—
Grouped	RESULT_GROUPED_AGG_CHUNKs of occupied 64-byte group slots + EXEC_SUCCESS	1,023 slots

Every chunk names the container its offsets index (container_id in the header), because a worker walks every container of a table and a row offset means nothing without its container.

Streaming is one mechanism everywhere: the VM stops when the chunk buffer fills, the worker flushes, the VM resumes from the reported row. A 100 M-row answer needs the same two fixed buffers as a 10-row one.

Safety valves

• A program that reads a bitmapped (nullable) column without the NULLABLE flag is refused with EXEC_ABORT — the dense read would mis-key every row after the first NULL and still return plausible integers.
• A HASH_ACC program whose group table fills is refused, never merged.
• An unknown table, an unresolvable column, a malformed join table, an append at the wrong offset — all EXEC_ABORT, all counted.
• A request for a tenant other than the system tenant on a worker without AETHERIUS_ENABLE_MULTI_TENANCY=1 is aborted before any file is resolved, and counted — the worker does not rely on the orchestrator having checked.
• A DELETE or UPDATE that meets a row held by another in-flight transaction answers WRITE_CONFLICT instead of waiting.

What a worker keeps on disk

• One directory per tenant. The system tenant uses the data directory itself; tenant 7 lives under tenant_7/. A statement for a tenant only ever opens that tenant’s files. See Multi-Tenancy.
• Sidecars beside each container. Deleted-row marks (.tomb), secondary indexes (.idx) and long-string data (.str) sit next to the .acf they belong to and move, link and vanish with it.
• Snapshots under snapshots/tenant_<t>_<id>/: hard links of everything the tenant owned at that moment, taken in the time it takes to link the files.
• The vacuum. A background sweep every AETHERIUS_VACUUM_INTERVAL seconds (default 60) rewrites files that are more than half dead and removes files left by rolled-back transactions, never blocking a reader or an insert. See Deletes, Updates & the Vacuum.
• New files are noticed as they land. A container shipped by an INSERT, a snapshot directory, or a tenant directory that did not exist at boot is picked up on the next statement that needs it.

Counters

rows_yielded, chunks_sent, blocks_walked, refused, bytes_sent, probes_run, tables_loaded, segments_served, segments_appended, frames_received, stale_rejected. Tests assert these moved; the bytes_sent counter is how “80 bytes for a 10 M-row COUNT” is a measurement rather than a claim.

Concurrency

Per connection one thread; per worker one accept thread; no lock on any per-row path. The broadcast table and group table belong to the connection that loaded them. The segment fetch handler opens its own descriptor, so a shadow tail never contends with the primary’s append writer.

Cluster Execution Engine →

Orchestrator, worker and the path between them.

Tiny VM ISA →

The programs a worker executes.

Shadow HA Streamer →

The frames a primary serves and a shadow accepts.

Data Persistence Engine →

The .acf container format the worker reads in place.