Walleye docs

A cluster

Three nodes, each running the API and an embedded replica daemon. A write is acknowledged once a quorum of replicas holds it, so losing one node loses nothing that was acknowledged. It does cost service to the tables that node owned until it comes back, which losing a node is about.

What each node runs

Setting WALLEYE_BITR_URL turns on quorum durability and starts the replica daemon inside the same process. The daemon holds the write-ahead log on local disk, replicates it to its peers, and archives committed segments to object storage.

export WALLEYE_BUCKET=my-bucket
export WALLEYE_MEMBERS='a=http://node-a:8080,b=http://node-b:8080,c=http://node-c:8080'
export WALLEYE_NODE_ID=a
export WALLEYE_BITR_URL=http://127.0.0.1:30080
export WALLEYE_TOKEN=... # the same on every node

Membership is static and identical on every node. A stream’s owner is its rendezvous winner over that list, so ownership is the same on every node and survives a restart. A request that arrives at a non-owner is forwarded.

The replica daemon

It reads its own environment, and those names still carry a LAKEDAY_ prefix from where the code came from. That is a wart, not a hint that something managed is required.

VariableMeaning
LAKEDAY_DATAPLANE_ROOT_KEYRoot key the replicas authenticate with. The same on every node.
LAKEDAY_REPLICA_NODE_NAMEThis replica’s name.
LAKEDAY_REPLICA_MEMBERSThe replica peers, in the same form as the API members.
LAKEDAY_REPLICA_INTERNAL_TOKENShared between replicas.
LAKEDAY_REPLICA_DATA_DIRWhere the log lives, /data by default.
LAKEDAY_REPLICA_QUORUMHow many must hold a write, 2 by default.
LAKEDAY_REPLICA_ARCHIVE_BUCKETWhere committed segments are archived.
LAKEDAY_REPLICA_ARCHIVE_PREFIXPrefix within it.
LAKEDAY_REPLICA_ARCHIVE_BATCH_RECORDSRecords per archived segment.
LAKEDAY_REPLICA_TIERFree-text tier label carried in membership.

WALLEYE_DATA_KEY holds the 32-byte key, base64, that records are encrypted with before they leave the node.

Starting for the first time

A replica with an empty volume learns the archived prefixes before it serves. That seeding is retried rather than fatal, because a transient storage failure at boot would otherwise restart the node, fail again, and turn a passing fault into a crash loop. The listeners stay unbound while it retries, so the node never reports healthy while it cannot serve.

Checking it

curl -s localhost:8080/readyz -H "authorization: Bearer $TOKEN"
curl -s 'localhost:8080/readyz?require=all' -H "authorization: Bearer $TOKEN"

/readyz reports write readiness and names which members are healthy and which are unreachable. ?require=all asks the stricter question, which is the one a single address in front of the cluster needs to ask.

Readiness has hysteresis: a single slow probe does not withdraw write readiness, because a probe timing out under load is not the same as a quorum being gone.

Losing a node

Read this part carefully, because durability and availability come apart here.

Nothing committed is lost. A write was acknowledged only once a quorum held it, so the two surviving replicas have it and the archive receives it.

The tables that node owned stop answering. Membership from WALLEYE_MEMBERS is static, and a stream’s owner is a pure function of its name over that list with no liveness in it. A dead node therefore stays the owner of its streams. Requests for them are forwarded to it, retried for fifteen seconds, and then answered 502. Tables owned by the other two carry on untouched.

So a three-node cluster survives losing a node without losing data, and without losing service to roughly two thirds of its tables. It is not a failover cluster for the remaining third. Plan capacity and expectations on that basis, and bring the node back rather than waiting for the cluster to route around it.

Ownership does move when membership itself changes, which today means Kubernetes EndpointSlice discovery rather than the static list: the ring is rebuilt as endpoints come and go. That path is not configured by the environment variables above.

When it does move, it moves at once. There is a warming window after a membership change, but it applies to cached reads, which fall back to the previous owner on a miss. Stream ownership has no such grace: the new owner owns it from the moment the ring changes.

Restoring a node that lost its disk is the seeding path above: it reads the archive, catches up from its peers, and rejoins as the owner of the same streams it had before, because the list never changed.