Cluster Execution Engine
cluster. Flags and environment
variables cannot enable it. To license this feature, contact the
AetheriusDB team at
aetheriusdb.com or
aetheriuslabs.com.
The orchestrator and every worker need the grant; a worker started without it exits instead of serving.
See Licensing → Premium features.
What it is
Section titled “What it is”When AETHERIUS_CLUSTER_MODE=1 is set, the daemon becomes an orchestrator:
it owns the catalog and the SQL surface but holds no data. Each worker
(AETHERIUS_WORKER_MODE=<cluster>) owns a directory of .acf containers and
nothing else — no parser, no planner, no catalog. Between them runs a single
idea, applied everywhere:
The orchestrator compiles the question into a few dozen 64-bit instructions. The worker runs them over its column stripes in a register VM and sends back only what the question produces: row offsets, projected cells, one aggregate scalar, or one hash-table slot per group.
Nothing is parsed on the worker. Nothing is allocated per row on either side. The worker never sees SQL; the orchestrator never sees a row it did not ask for.
The pieces
Section titled “The pieces”| Piece | Where | What it does |
|---|---|---|
| Remote dispatcher | aetherius-runtime::dispatch::remote | Routes a statement to the cluster that owns its table (or the two clusters of a join), or refuses with a message that names the reason. |
| Bytecode compiler | aetherius-orchestrator | Turns WHERE, projections, aggregates, GROUP BY and join conditions into Tiny VM programs. Zero heap allocations; ~25 ns for a three-predicate WHERE. |
| Tiny VM | aetherius-runtime::vm | A 256-register interpreter with four specialised row loops (filter, materialize, aggregate, join). Reads column stripes in place, including packed nullable stripes through a rank dictionary. |
| Worker node | aetherius-runtime::worker | A raw TCP listener that discovers its containers, runs programs over every block of the target table, and streams chunked results. |
| Wire protocol | aether-cluster-wire | 64-byte header, fixed record payloads viewed in place on both sides. Version 17 as of this arc. |
| Shadow HA streamer | aetherius-runtime::dispatch::shadow_streamer | Asynchronous tail replication of a primary’s .acf to its standby, driven by heartbeat PONGs. |
What a query costs
Section titled “What a query costs”Measured on one Apple M-series core with criterion; each number states what
was measured and the caveat that applies. “Dense” means a column with no
NULLs; “1 M” is one million rows in a real .acf container.
| Operation | Measured | Caveat |
|---|---|---|
| VM dispatch, mock block | 0.78–0.98 ns per instruction | Varies per build (see codegen note below). |
Compile a 3-predicate WHERE | 25.5 ns, 13 instructions, 0 allocations | — |
WHERE col1 != 5 OR col2 <= 10, 1 M dense rows | 5.78–5.90 ms → 5.8 ns/row, 170 M rows/s | The standing regression guard for the filter loop. |
Nullable LOAD_COL through the rank dictionary, 1 M rows | 1.12 ns per instruction | — |
WHERE col1 > 60, 3 containers, 330,000 rows, 105,600 matches | 1.62 ms VM walk; 3.22 ms end to end over loopback TCP, 27 chunks | Includes result materialisation at the orchestrator. |
SELECT COUNT(*) … WHERE over 10,000,000 rows | 80 bytes on the wire, one frame | Offsets would have been 160 MB in 2,443 frames. |
SELECT SUM(price) WHERE qty > 0, 5,000,000 rows, 5 containers | 27.7 ms end to end (5.5 ns/row), 80 bytes of worker output | Offsets would have been 72 MB. |
| Bytecode join, 20,000 probe rows × 3 build keys, 2,880 pairs | rows identical to the oracle | Correctness witness, not a timing. |
SQL JOIN across two workers, 30,000 × 2,000 rows | rows identical to the oracle | Through the real dispatcher from SQL text. |
The codegen note
Section titled “The codegen note”The VM’s row loop is sensitive to how many arms it carries. Adding the aggregate and projection arms to one monolithic loop moved the dense 1 M filter bench from 5.90 to 7.3 ms; four structurally equivalent variants measured between 5.8 and 8.2 ms with run-to-run spread under 1 percent. An aligned-loops build reproduced the numbers to 0.1 ms, so alignment was ruled out: the cause is LLVM register allocation across one large match function spilling to the stack. The fix that held was one loop per program kind, each holding only the arms it can execute. The dense bench is back to 5.8 ms and is re-run after every change to the loop.
Observability
Section titled “Observability”Every branch that can fire exposes a counter, and the tests assert the
counter moved rather than that wall time improved. The system.execution_mode
view reports the dispatcher’s mode and its counters:
| Column | Meaning |
|---|---|
filters_compiled | Programs compiled at the seam. |
frames_shipped | EXEC_BYTECODE_REQ frames sent to workers. |
chunks_received | Result frames consumed (offsets, cells, aggregates, groups). |
remote_rows | Rows materialised from worker replies. |
Worker-side counters (worker_exec, worker_table, worker_segment) count
rows yielded, chunks sent, blocks walked, refusals, bytes sent, probe frames
run, tables loaded, segments served and appended.
Environment variables
Section titled “Environment variables”| Variable | Role | Effect |
|---|---|---|
AETHERIUS_CLUSTER_MODE=1 | Orchestrator | Selects the remote dispatcher at boot. Decided once, cached for the process. |
AETHERIUS_TOPOLOGY_JSON / AETHERIUS_TOPOLOGY_FILE | Orchestrator | The cluster topology: clusters, their addresses, the tables they own, an optional standby. |
AETHERIUS_WORKER_MODE=<cluster> | Worker | Boots as a worker for the named cluster. No SQL surface is bound. |
AETHERIUS_WORKER_PORT | Worker | Listening port. Default 5434. |
AETHERIUS_WORKER_DATA_DIR | Worker | The containers/ directory to discover blocks from. Unset: the worker serves no data and answers every frame EXEC_SUCCESS. |
AETHERIUS_ENABLE_MULTI_TENANCY=1 | Both | Accept tenants other than the system tenant. Off by default; see Multi-Tenancy. |
AETHERIUS_MAX_CONCURRENT_QUERIES_PER_TENANT | Orchestrator | Statements one tenant may have in flight. Default 50. |
AETHERIUS_TX_TIMEOUT | Orchestrator | Milliseconds before a stalled transaction is aborted. Default 5000. |
AETHERIUS_VACUUM_INTERVAL | Worker | Seconds between vacuum sweeps. Default 60. |
What runs in the VM today
Section titled “What runs in the VM today”- Filters:
=,!=,<,<=,>,>=,AND,OR,NOT,+,-,*,/over integer columns and literals. Division by zero yields NULL. - Projections: any list of columns or
*, streamed as columnar chunks with validity bitmaps. - Aggregates:
COUNT,SUM,MIN,MAX,AVG(asSUMandCOUNT, divided at the orchestrator), with or withoutWHERE. - GROUP BY one column with up to four accumulators per group (AVG uses two).
- Joins:
INNERandLEFTequi-joins on one or more columns, with build-side columns projected from the broadcast table and cross-sideWHEREconjuncts evaluated after the probe. - Text: VARCHAR columns —
=,<>,<,<=,>,>=andLIKE 'prefix%', selected back as text. See VARCHAR in Cluster Tables. - Writes:
INSERT … VALUES,DELETE … WHERE,UPDATE … SET … WHERE, inside or outside a transaction. See Deletes, Updates & the Vacuum and Cluster Transactions. - Indexes: point lookups, ranges and prefix searches through a secondary index. See Secondary Indexes on Cluster Tables.
What does not, yet
Section titled “What does not, yet”Recorded in the plan documents, not hidden:
RIGHTandFULLjoins; joins of more than two tables.- Floats and timestamps in the VM (they route to the single-node executor); text takes no part in joins.
GROUP BYon more than one column; more than four accumulators per group; group tables above 4,096 slots are refused rather than spilled.- Cluster mode refuses DDL, and the catalog must exist before the daemon boots as an orchestrator.
SET,SHOW,BEGIN/COMMIT/ROLLBACK,EXPLAIN,SELECTwithout aFROMand thesystem.*views run on the orchestrator locally and are counted. CREATE INDEX, per-tenant schema changes, tenant snapshots and restores are orchestrator API calls, not SQL statements.
How it was built
Section titled “How it was built”The arc landed in thirteen phases over one working day, each a set of small commits behind a file-length gate and a benchmark. The phase log is in Engine Hardening Log.