Table Clustering & the God Cluster

DistributedClustersJoinsPerformance

Beta Works · surface still evolving · Table-to-cluster assignment, co-location relations, per-cluster writes, same-worker and cross-cluster joins, and the God Cluster (Broadcast fan-out writes, local joins, read fallback) all run through the real dispatcher against real workers. Freshness and the size cap are process-local — see the limits.

Premium feature Available only with a license key that grants cluster. Flags and environment variables cannot enable it. To license this feature, contact the AetheriusDB team at aetheriusdb.com or aetheriuslabs.com. The orchestrator and every worker need the grant. See Licensing → Premium features.

What it is

Table clustering is how you tell AetheriusDB which cluster owns which table. Every table belongs to exactly one cluster. That cluster runs the dedicated write process for its tables; nothing else writes them. Reads are not confined the same way — a query may touch tables in several clusters, and the orchestrator routes each part to the cluster that holds it.

The God Cluster is the cluster for tables every other cluster needs — users, roles, regions, lookup and configuration tables. Its tables are replicated to every worker, so a join against them never leaves the worker that runs it.

The one rule that decides whether a clustered deployment is fast:

Tables that join each other live in the same cluster. A join inside one cluster runs on one worker with no network. A join across two clusters works, but it is the fallback for a schema that was not clustered along its relationships — not the path to design for.

Why it matters

Writes scale with clusters. Each cluster has its own write process on its own machine. Adding a cluster adds write capacity instead of sharing one node’s.
Joins stay local. Related tables in one cluster join on that cluster’s worker; nothing crosses the wire but the answer.
Failures stay contained. A dead orders cluster refuses orders queries; customers keeps answering.
Shared reference data is everywhere it is needed (God Cluster, once wired): one write, present on every worker.

How you use it

1. Group tables by relationship

Start from your joins, not your table sizes. For each table, ask: what does it get joined with? Put those tables together.

orders_domain            customer_domain          products_domain
├── orders               ├── customers            ├── products
├── order_items          ├── addresses            ├── inventory
└── shipments            └── payment_methods      └── categories

god_cluster   (shared — everything joins these)
├── users
├── roles
└── regions

Checks before you commit to a layout:

Every foreign key that a query follows should stay inside one cluster.
A table that many clusters join, and that is small, is a God Cluster candidate. A table that many clusters join and that is large does not belong in the God Cluster — put it with the cluster that joins it most.
If two big tables in different clusters must be joined, that join will run as a cross-cluster join. It is correct; it is just not local.

2. Write `topology.json`

Membership is declared in a file the orchestrator reads at boot. It is not derived from foreign keys, so adding a constraint later never silently moves a table between clusters.

{
  "clusters": {
    "god_cluster": {
      "addr": "10.0.0.1:5433",
      "standby": "10.0.0.11:5433",
      "strategy": "broadcast",
      "tables": ["public.users", "public.roles", "public.regions"]
    },
    "orders_domain": {
      "addr": "10.0.0.2:5433",
      "standby": "10.0.0.12:5433",
      "tables": ["public.orders", "public.shipments"]
    },
    "customer_domain": {
      "addr": "10.0.0.3:5433",
      "tables": ["public.customers", "public.addresses", "public.payment_methods"]
    },
    "products_domain": {
      "addr": "10.0.0.4:5433",
      "tables": ["public.products", "public.inventory", "public.categories"]
    }
  },
  "relations": [
    "public.order_items.order_id -> public.orders.order_id"
  ]
}

Key	Required	Meaning
`addr`	yes	The cluster’s worker, `host:port` or `IP:port`. Hostnames are resolved once, at boot.
`standby`	no	A second node that receives an asynchronous copy of the cluster’s containers and is the failover target. Without one, a dead cluster refuses rather than failing over.
`strategy`	no	`"local"` (default): the tables live on this worker only. `"broadcast"`: every write is copied to every worker — this is the God Cluster.
`tables`	yes	The tables this cluster owns.
`relations`	no	Top level. `"child.col -> parent.col"` places the child table in the parent’s cluster, so a child row always lands on the worker holding its parent. A child named here needs no `tables` entry; naming it under a different cluster fails the boot. Chains (`items → orders → users`) may be written in any order.

Qualify table names. public.orders means “the orders table in schema public”; a bare orders means “any orders in any schema of any database”. If two databases or schemas both have an orders, a bare key routes one of them to the wrong cluster and the query still looks like it worked. Write db.schema.table when you have more than one database.

A table may appear in one cluster only. Listing it twice, or both bare and qualified, fails the boot with the table’s name in the message.

The orchestrator finds the file in this order: AETHERIUS_TOPOLOGY_JSON (the document inline), then AETHERIUS_TOPOLOGY_FILE, then <data_dir>/topology.json.

3. Start one worker per cluster

Each cluster is an aetheriusd process in worker mode. It owns a data directory and serves only the tables in it; it has no SQL surface of its own.

# god_cluster
AETHERIUS_WORKER_MODE=god_cluster \
AETHERIUS_WORKER_PORT=5433 \
AETHERIUS_WORKER_DATA_DIR=/data/god_cluster \
aetheriusd

# orders_domain
AETHERIUS_WORKER_MODE=orders_domain \
AETHERIUS_WORKER_PORT=5433 \
AETHERIUS_WORKER_DATA_DIR=/data/orders \
aetheriusd

Repeat for every cluster named in the topology, on the host the topology’s addr points at. A worker started without the cluster license grant exits instead of serving.

4. Start the orchestrator

The orchestrator is the process clients connect to. It owns the catalog and the SQL surface, holds no table data, and routes every statement to the cluster that owns its table.

AETHERIUS_CLUSTER_MODE=1 \
AETHERIUS_TOPOLOGY_FILE=/etc/aetherius/topology.json \
aetheriusd --data-dir /var/lib/aetherius

Cluster mode is decided once at boot. Changing the topology — adding a cluster, moving a table — means editing the file and restarting the orchestrator.

5. Confirm what the orchestrator decided

SELECT * FROM system.execution_mode;

mode should read remote, topology_installed should be true, and topology_clusters / topology_tables should match your file. If the file failed to load, topology_error says why — the daemon still starts and refuses routable statements so you can read this view.

6. Write and read through the orchestrator

-- Routed to orders_domain; its write process appends the row.
INSERT INTO orders (order_id, customer_id, amount) VALUES (1001, 42, 250);

-- Routed to orders_domain; the join runs on that worker, no network.
SELECT o.order_id, oi.sku, oi.qty
FROM orders o
JOIN order_items oi ON oi.order_id = o.order_id
WHERE o.amount > 100;

-- Two clusters. Works, as a cross-cluster join: the orchestrator counts
-- both sides, ships the smaller side to the larger side's worker, and
-- probes there. Correct, but not local — this is the shape to design away.
SELECT o.order_id, c.email
FROM orders o
JOIN customers c ON c.customer_id = o.customer_id
WHERE c.tier > 1;

Cross-cluster joins are visible as cross_joins_planned in system.execution_mode. If that counter climbs on your everyday queries, the tables in those queries belong in one cluster.

Large joins: the radix shuffle

The cross-cluster join above ships the smaller side whole to the larger side’s worker. That is the right move while the smaller side is at most 10 MB. When both sides are larger, the orchestrator switches to a radix shuffle: every row of both tables is sent to the worker chosen by a hash of its join key, so matching rows meet on one node and each worker joins its own slice. The decision is made from measured row counts, not guessed.

-- Two million-row tables, neither side small enough to ship whole.
SELECT COUNT(*) FROM orders o JOIN customers c ON o.cid = c.id;

What to know before relying on it:

It answers COUNT(*). The shuffle route serves SELECT COUNT(*) over a single-key INNER JOIN with no predicate that spans both tables. Any other shape falls back to the broadcast route, which answers or errors — never silently.
The build side’s key must be unique (a primary key, in practice). The orchestrator checks this before shipping a byte and declines otherwise, because the worker keeps one row per distinct key and would undercount.
You need enough workers. Each worker holds at most 65,536 rows per table for one shuffle, so a side of R rows needs at least R ÷ 65,536 active workers, up to 16. A million-row side needs all sixteen. Clusters with no tables of their own ("tables": []) count as workers.
Watch it: shuffle_joins, shuffle_declines, shuffle_data_frames and shuffle_rows_sent in system.execution_mode. A declined large join leaves its reason readable to the next EXPLAIN-style surface; today shuffle_declines climbing with shuffle_joins flat means the query shape or the fabric size is the blocker.

Transactions that write to more than one cluster

Inside BEGIN … COMMIT, every cluster a statement writes to becomes a participant. COMMIT then runs in two phases before the transaction becomes visible:

Prepare — each participant’s active worker is asked whether it still holds the transaction’s writes. A worker that has failed over to a standby which never received them says no.
Decide — if every participant said yes, all are told to commit and the transaction becomes visible everywhere at once. If any said no (or could not be reached), all are told to abort, the transaction is rolled back, and COMMIT returns an error asking you to retry (SQLSTATE 40001).

ROLLBACK tells every participant to abort as well, and a worker that is told to abort releases the rows the transaction had claimed immediately — another writer can take them at once. Before this, rows a rolled-back or stalled transaction had touched stayed locked until the background vacuum happened to notice.

BEGIN;
UPDATE orders    SET status = 'shipped' WHERE order_id = 1001;   -- orders_domain
UPDATE customers SET last_order = 1001  WHERE customer_id = 42;  -- customer_domain
COMMIT;   -- asks both workers, then commits both, or rolls back both

Watch it in system.execution_mode: two_phase_commits, two_phase_aborts, prepares_sent, prepare_aborts. A rising prepare_aborts means a participant is answering from an endpoint that does not hold the writes — usually a failover onto a standby that is behind.

Moving a cluster, and fencing off whoever missed the move

A cluster can be moved to another worker while the orchestrator is up. Every move is two things at once: the routing pointer changes, and the manifest epoch rises. Every frame the orchestrator sends carries that epoch, and a worker remembers the highest one it has seen — so an orchestrator still holding the old routing (a stale instance, a split brain) is refused by the workers from the moment of the move, not discovered later as divergent data.

Three operator actions, all through the orchestrator’s admin API (aetherius_orchestrator::topology):

Action	What it does
`publish_override(cluster, target)`	Route the cluster to `target` now; bump the epoch. For a target you prepared out of band (a copy already in place).
`clear_override(cluster)`	Route back to the topology primary; bump the epoch.
`migrate_cluster(cluster, target, source)`	The four-phase live migration: copy base segments, copy the delta, close the cluster’s write gate, drain in-flight writes, swap the pointer, reopen, then release the source. Writers are paused only for the swap; reads never stop.

The override outranks a standby: a cluster that is both mid-migration and failed over routes to the migration target, because the standby holds the old data.

Strict fencing. By default a worker accepts frames that carry no epoch, so single-node use and older tools keep working. Set AETHERIUS_WORKER_STRICT_FENCING=1 on a worker to refuse them: from then on only an orchestrator publishing a manifest can drive that worker, and a stale orchestrator’s frames are answered EXEC_ABORT immediately. Turn it on once every orchestrator on the fabric runs cluster mode — a fabric with one unfenced sender left would go dark. Watch manifest_epoch, route_overrides_active and migrations in system.execution_mode, and the worker’s stale_rejected count.

The God Cluster

Beta Works · surface still evolving · Declared with strategy: broadcast. Writes fan out to every worker and are acknowledged; joins against a Broadcast table run on the other side's worker when that worker is in sync; a read survives a dead owner via an in-sync copy. Freshness and the 256 MB cap are in-memory and reset at every orchestrator boot.

What it is for

Some tables are joined by everything: users, roles, regions, feature flags, currency tables. Put one of those in an ordinary cluster and every other cluster’s queries pay a cross-cluster join to reach it.

The God Cluster is the cluster declared "strategy": "broadcast": every INSERT, UPDATE and DELETE on any of its tables is shipped to every worker in the deployment and acknowledged before the statement returns. After that, a query on orders_domain that joins users finds users already on its own worker — the whole join runs there. The same holds from customer_domain, products_domain, and any cluster you add later.

How it behaves

The declaration is per cluster, not per table. Every table in the God Cluster is broadcast; no table in a Local cluster is.

Writes go everywhere, and the owner must answer. The God Cluster’s own worker must acknowledge or the statement fails. Any other worker that does not acknowledge in time is marked out of sync and the statement still succeeds — you see it as broadcast_targets_degraded in system.execution_mode.
A stale worker is never used for a local join. A worker that missed a broadcast keeps getting the two-worker cross-cluster join until it acknowledges the next one. Slower, never a short answer.
Reads prefer the owner, and survive it. A single-table SELECT on a God Cluster table goes to the God Cluster’s worker; if that worker is dead and has no standby, any in-sync worker answers instead (broadcast_read_fallbacks).
Keep it small. A broadcast write costs one network round trip and one disk write per worker, so the cluster carries a hard ceiling: 256 MB across all of its tables together, not per table. The append that would cross the line is refused with the current size in the message.
Expect writes to be slower than reads. The right trade for reference data that changes rarely and is read constantly; the wrong home for anything written continuously.

Watching it work

SELECT broadcast_fanouts, broadcast_targets_acked, broadcast_targets_degraded,
       broadcast_local_joins, broadcast_read_fallbacks,
       local_joins_planned, cross_joins_planned, topology_relations
FROM system.execution_mode;

broadcast_local_joins climbing while cross_joins_planned stays flat is the God Cluster doing its job. local_joins_planned counts joins between two tables on one worker — co-located by tables or by relations.

What is not supported yet

Live topology reload. The file is read once. Adding or moving a cluster or a table is a restart, and a restart can take minutes while indexes rebuild.
DDL in cluster mode. CREATE TABLE, CREATE INDEX via SQL and the rest are refused by the orchestrator; the catalog must exist before it boots as an orchestrator. Index builds are an orchestrator API call.
Durable freshness and size accounting. Which workers hold the latest broadcast, and how much the God Cluster holds, are tracked in the orchestrator’s memory. After an orchestrator restart every join against a Broadcast table takes the two-worker path until the next broadcast write, and the 256 MB cap counts from zero. A DELETE does not give bytes back to the cap until this is made durable.
Per-table replication. Strategy is a cluster property. A table cannot opt in or out of its cluster’s strategy. A relation cannot point at a Broadcast parent — the parent is on every worker, so “the parent’s node” names nothing; place such a child in a Local cluster.
More than two tables in one cross-cluster join, and RIGHT/FULL joins across clusters.
Shuffle joins that return rows. The radix shuffle returns a count; projections over two large tables take the broadcast route and its limits. Rows are pulled to the orchestrator and pushed to the workers one worker at a time.

Troubleshooting

You see	It means	Do this
`cannot determine a routing table for this statement`	A DDL or session statement reached the router.	Run DDL before booting as an orchestrator; session statements (`SET`, `BEGIN`, `SHOW`) are handled locally and should not produce this.
`not routed to a cluster node` / `table not owned by any cluster`	The table is in no cluster’s `tables` list, or is listed under a different qualification than the query used.	Add it to `topology.json`, qualified the way your queries name it, and restart the orchestrator.
`the cluster topology failed to load (...)`	Malformed address, a table claimed twice, or a bad key. The message names the offender.	Fix the file; the daemon is up and will serve once restarted. Set `AETHERIUS_REQUIRE_TOPOLOGY=1` if you would rather it refuse to start.
`cross_joins_planned` rising on routine queries	Related tables are split across clusters.	Move them into one cluster (a `relations` entry does this for a child), or into the God Cluster if they are small and widely joined.
`broadcast_targets_degraded` rising	A worker keeps missing broadcasts (slow disk, network, or down without the monitor noticing yet).	Check that worker; until it acknowledges a broadcast, joins against God Cluster tables from it run cross-cluster.
`relation … parent … is not declared` at boot	The parent table in a relation is in no cluster’s `tables` and no other relation places it.	Declare the parent, or fix the name; keys must match the declaration exactly.
A whole cluster’s queries refuse after a node dies	The cluster declared no `standby`.	Add one and restart; without a standby a dead cluster refuses instead of failing over.