Walleye docs

Shapes

A Walleye deployment is one node or three. That is the shape, and it is the only structural choice: the binary, the protocol and the guarantees do not change with it.

One nodeThree nodes
A write is durable whenthe log reaches object storagea quorum of replicas holds it
One node is lostnothing answers until it is backno acknowledged write is lost; the tables it owns stop answering until it is back
Ownershipeverythingeach table belongs to one node
Turned on bynothing, it is the defaultWALLEYE_MEMBERS and WALLEYE_BITR_URL

A single node and a cluster are how to run each. This page is what changes between them.

Any node, any request

Every member serves the whole API, so a client needs no knowledge of the topology and nothing pins it to a node.

Each table is owned by exactly one member, picked by weighted rendezvous hashing over the member list. Only the owner holds the writer, so a request that arrives anywhere else is forwarded to the owner and answered from there. Reads are forwarded too, which is why a read always includes rows that are still in memory.

Membership is static and identical on every node, so ownership is the same everywhere and survives a restart.

Losing a node

Read this part before you size a cluster on it, because durability and availability come apart here.

Nothing committed is lost. A write is acknowledged only once a quorum of two replicas holds it, so the two survivors hold everything that was acknowledged, and the archive receives it behind them.

The tables that node owns stop answering. Ownership is a pure function of the table name over the static member list, with no liveness in it, so a node that is away stays the owner of its tables. Requests for them are forwarded to it, retried for fifteen seconds, and then answered 502. Tables owned by the other two carry on untouched.

So a three-node cluster survives a node loss without losing data and without losing service to roughly two thirds of its tables. It is not a failover cluster for the remaining third. Bring the node back rather than waiting for the cluster to route around it.

Ownership does move when membership itself changes, which today means Kubernetes endpoint discovery rather than the static list. WALLEYE_MEMBERS does not turn that on. A membership change moves stream ownership at once; the warming window that follows one applies to cached reads, not to which node owns a table. A cluster is where that lives.

Fencing

Ownership does not move under a node loss, so fencing is not about failover. It is about the paths that do hand a table’s writer from one process to another — a restart, a reopen, an ownership change under endpoint discovery — because two writers for one table would be a lost-update machine. A new owner claims the next writer epoch through a compare-and-swap on the manifest, and from that moment the previous owner’s appends and commits fail. In a cluster the replicas are fenced from the same claim, so a process that was slow rather than dead cannot write behind the new owner’s back.

Forwarded requests carry the sender’s view of the membership and are refused if the receiver’s differs, and a forward that lands on a non-owner is refused rather than forwarded onward.

SQL across owners

A SELECT whose tables all share one owner is forwarded there and runs there. A SELECT that spans owners works too: the node that received it gathers the rows it does not own from their owners and runs the query locally. What it gathers is that owner’s own snapshot, so it carries unflushed rows and the answer is not stale.

Gathering has a ceiling. A table with more than a million rows is refused rather than shipped across the network:

stream <name> has more than 1000000 rows; query it on its owner rather than
joining it across members

The limit is on how many rows one table has to move — not on how large a table may be, and not on how many a statement may name. The fix is to stop moving them: query that table on its own, or alongside only tables its own node owns, and the whole statement is forwarded instead of gathered. If you need the join, narrow the large side first.

Readiness

Two probes, because a cluster makes two different questions out of one.

/healthz reports unavailable until a node can make writes durable — a reachable quorum, and its own tables opened. Point a load balancer at it and traffic never reaches a node that would refuse a write. Once a node has served it stays healthy even if the quorum is later lost, because reads stay correct without one.

/readyz is the strict question: can this node make a write durable. It names the members serving and the members it cannot reach, so a caller that reaches the cluster through one address can tell a whole cluster from a quorum of one. /readyz?require=all is stricter still and reports ready only when every configured member is serving, which is what “the cluster is up” means to a client about to write to a table the missing member owns.

It is not asked at the moment you ask it. A background poll refreshes it every two seconds, and write readiness is withdrawn only after three consecutive failed polls, so /readyz can answer 200 for about six seconds after a quorum is gone — longer if the polls are timing out rather than failing fast. That hysteresis is deliberate: a probe timing out under load is not the same as a quorum being gone. Treat a ready answer as “ready a moment ago”, and do not build a fence out of it that assumes otherwise.

Both the member names and ?require=all come from the replica gateway’s own answer. A node running without WALLEYE_BITR_URL — a single node — answers a literal {"ready": true}: it names nobody, and require=all is a silent no-op rather than an error.

A write that arrives while the quorum is unreachable gets a 503. On the LanceDB routes it carries Retry-After; the native write routes return a bare 503 with the reason in the body. A forward to a peer that is still starting is retried for up to fifteen seconds. A member that takes longer than that to boot is better gated on with require=all than waited out.

On the managed service

ramp is one node and launch is three, provisioned with the replica daemon already wired up. Moving between them is a resize, and it is one way: an instance that has run as launch does not go back to ramp.