Multi-Tenancy (Fluid Clusters)

DistributedMulti-TenancySecurity

Beta Works · surface still evolving · Tenant routing, isolation, quotas, snapshots and restore are proven end to end. Every client protocol — native, PostgreSQL wire and Flight SQL — now binds the tenant at connection time; DROP TENANT is SQL. Snapshot and restore remain orchestrator API calls.

Premium feature Available only with a license key that grants multi_tenancy. Flags and environment variables cannot enable it. To license this feature, contact the AetheriusDB team at aetheriusdb.com or aetheriuslabs.com. Set on the orchestrator and on every worker. See Licensing → Premium features.

What it is

A cluster can serve many tenants at once. Each connection executes for exactly one tenant, every statement it sends is routed for that tenant, and on the worker each tenant’s rows live in a directory of their own. A tenant never sees, counts or deletes another tenant’s rows — not through a query, not through an index, not through a backup.

Tenant 0 is the system tenant. It maps to the worker’s root data directory, so a single-tenant deployment needs no configuration at all.

Turning it on

Multi-tenancy is off by default. Turn it on for the orchestrator and for every worker, with either the daemon flag or the environment variable:

aetheriusd --enable-multi-tenancy          # or:
AETHERIUS_ENABLE_MULTI_TENANCY=1 aetheriusd

The flag is also an install-record field (enable_multi_tenancy under [server]), so a supervised service carries it into its generated systemd unit or launchd plist. The flag only ever turns the switch on; it and the environment variable compose rather than override each other.

With the flag off, any statement or request for a tenant other than 0 is refused before anything is routed or read: the client gets a TenancyViolation error (SQLSTATE 08P01), and a worker that receives such a request directly aborts it. Both ends check independently, so a worker never relies on the orchestrator having done so. Every refusal is counted.

How a connection gets its tenant

The tenant is a property of the socket, never of the SQL. It is settled during the connection handshake, before the session exists and before the connection is told it may send a statement. It is then immutable for the life of that connection, and no statement can change it.

Each protocol carries it the way that protocol already carries connection parameters:

Protocol	How the client names its tenant
Native	`AUTH_TENANT_REQ` as the first frame: `[tenant u64 LE][auth token…]`
PostgreSQL wire	a startup option — `PGOPTIONS="-c tenant_id=42"`, or `options=-c tenant_id=42` in the connection string
Flight SQL	the gRPC metadata header `x-aetherius-tenant-id: 42`

# psql, bound to tenant 42 for the life of the connection
PGOPTIONS="-c tenant_id=42" psql -h db.internal -U alice aetherius

With multi-tenancy on, a connection that names no tenant is refused rather than defaulted to the system tenant — defaulting is the whole hole, because it hands tenant 0’s data to a client that never asked for it. The pgwire refusal is a FATAL ErrorResponse (SQLSTATE 08P01) sent before ReadyForQuery, so the driver reports a tenancy error rather than a dropped connection. With multi-tenancy off, the option is optional and everything is tenant 0, so existing clients are unaffected.

Nothing in the SQL text names the tenant, so a query cannot be rewritten to reach another tenant’s data.

Flight SQL serves only the system tenant

Flight is gated like the others, but it refuses a tenant other than 0 rather than serving it. gRPC gives the service no connection-open hook, so there is nowhere to bind a tenant for the life of a Flight socket, and a per-query session cannot carry one. The refusal names the two protocols that can. This is a real limit, not an oversight: the alternative would look like isolation without being it.

Row-level security uses the same tenant

An RLS policy written against current_setting('app.tenant') resolves to the socket’s tenant, because the engine writes that variable when it binds the connection. A client SET app.tenant = … is refused with SQLSTATE 42501.

That is what makes a policy like this safe rather than advisory:

CREATE POLICY tenant_isolation ON orders
  FOR SELECT
  USING (tenant_id = current_setting('app.tenant'));

Without the binding, a client could SET its way to any tenant and the policy would filter faithfully against a value the client picked. See RBAC & Row-Level Security.

Where a tenant’s data lives

On a worker, tenant 7 reads and writes under tenant_7/ inside the data directory; the system tenant uses the data directory itself. An INSERT for a tenant lands in that tenant’s directory as a new container, and a SELECT only ever opens the blocks in the connection’s own directory. Two tenants inserting at the same time never touch the same file.

Noisy neighbours

One tenant cannot occupy the cluster. Each tenant may have a bounded number of statements in flight at the orchestrator:

AETHERIUS_MAX_CONCURRENT_QUERIES_PER_TENANT=50   # the default

A statement over the limit is refused immediately with a RESOURCE_EXHAUSTED error and never reaches a worker; other tenants are unaffected. The slot is released when the statement finishes, fails, or the connection drops, so a crashed client cannot pin a tenant’s quota.

Per-tenant columns without downtime

A cluster table’s schema can grow without rewriting a single file:

Add a column with a default — every existing row reads the default until the column is physically written for that tenant. SELECT and WHERE use it like any other column.
Add a nullable column — existing rows read NULL.
Per-tenant overlays — a column that exists for one tenant only, invisible to every other tenant of the same table.
Materialise — once a tenant’s new data carries the column physically, mark it materialised for that tenant; other tenants keep the default.

These are declared through the orchestrator’s schema API (aetherius_orchestrator::catalog::TableSchema, installed with catalog::install), not through SQL in this release.

Retiring a tenant

DROP TENANT retires a tenant’s silo, and it never blocks:

DROP TENANT 42;
DROP TENANT IF EXISTS 42;

The worker performs one rename(2), moving tenant_42/ into its .revert/ archive and stamping it with the drop’s transaction id. That is a single directory-entry update — constant time, whatever the tenant’s size — and the silo is unreachable by name the instant it returns. A synchronous recursive unlink would instead hold the connection for as long as the tenant was large, which is unbounded and proportional to somebody else’s data volume.

The bytes are reclaimed afterwards by the Reaper, incrementally on its own thread, and only once the visibility floor has passed the drop’s transaction id — so a query holding an older snapshot can still finish. An archive whose stamp cannot be read is held rather than guessed at.

Two rules the statement enforces:

Only the system tenant may issue it. A tenant is a silo, not an account with authority over other silos, so a connection bound to tenant 42 cannot drop tenant 7 — or itself, while it is still executing out of it.
Tenant 0 can never be dropped. Its directory is the data directory, and every other tenant’s silo is a subdirectory of it. Refused at the dispatcher and again at the worker.

DROP TENANT needs a worker. On a single-node process it is refused with a message saying so, rather than reporting success having deleted nothing.

Single-node deployments

Multi-tenancy is a cluster feature. On a single-node process there is one physical silo — the boot data directory — and no per-tenant directory to route to, because those are created by workers. So a statement that reads or writes table data for a non-system tenant is refused there rather than served from tenant 0’s tables.

Statements with no tenant component — SET, SHOW, BEGIN, SELECT 1 and the system.* views — are still served, so a bound connection can always be introspected and can answer a health check.

If you enable multi-tenancy on a single-node daemon and see refusals on ordinary queries, this is why: run the tenant through a worker (AETHERIUS_CLUSTER_MODE), or connect as the system tenant.

Snapshots and constant-time restore

A tenant snapshot is taken on the worker in one request and is complete in the time it takes to link the tenant’s files — it does not copy bytes, so a terabyte tenant snapshots in milliseconds. Every file the tenant owns is included: its containers and all of their sidecars (deleted-row marks, indexes and string data). A snapshot is immutable; asking for the same snapshot id twice is refused.

A restore is a routing change, not a copy: from the next statement on, every read and write for that tenant is served from the snapshot. Restoring to snapshot 0 returns the tenant to its live data. Nothing is moved and nothing waits.

Both are orchestrator API calls in this release (aetherius_orchestrator::topology::snapshot_tenant and restore_tenant_snapshot).

What is counted

Every tenancy counter is queryable as a virtual table:

SELECT subsystem, metric, value FROM system.tenant_metrics;

One row per counter, across five subsystems: the deployment gate (refused_dispatches, refused_frames, multi_tenancy_enabled), session binding (binds, refused_sets), the single-node guard (refused_statements), the noisy-neighbour quota (admissions, rejections, limit), and worker-side discovery (tenants_discovered, tenant_rediscoveries, snapshot_restores).

These are process-local counters, not a cluster roll-up: query the node you are asking about. On an orchestrator the worker rows read that process’s own worker-side state and are zero unless it is also serving as a worker. See Observability.

Limits in this release

Flight SQL serves only the system tenant — it gates the header and refuses any other tenant, because gRPC offers no connection-scoped session to bind one to.
Schema changes, snapshots and restores are API calls, not SQL. Only DROP TENANT has a SQL surface.
DROP TENANT addresses a worker directly, not through the cluster router — the statement names no table to route by. Issued as SQL in cluster mode it is currently refused with “cannot determine a routing table”; the orchestrator API path is what works today.
A tenant’s DDL still writes the shared catalog. There is one catalog per process, so CREATE/DROP TABLE from a bound tenant is visible to every tenant of that node. Per-tenant catalogs are a separate build-out; row and file isolation are unaffected.
Per-tenant SAL ingest budgets (AETHERIUS_SAL_TENANT_BUDGETS) are applied at boot but cannot bite on a single-node process, where the ingest producer runs as tenant 0.
A worker discovers each tenant’s directory lazily on first use; a directory created by hand is picked up on the next statement for that tenant.

Cluster Execution Engine →

What runs where, and the environment variables.

Cluster Transactions →

BEGIN, COMMIT, conflicts and timeouts across the cluster.

Deletes, Updates & the Vacuum →

How dead rows are reclaimed without blocking anyone.

Worker Node →

What a worker keeps on disk for each tenant.