Stop, start and resize

Stop

POST /v1/organizations/:org/instances/:id/stop, or Stop on the instance page, answers 202 with the record and drains the instance:

1. Each Machine gets SIGTERM with a 120-second grace period.
2. walleye-node quiesces: no new work is accepted, in-flight commits finish, and every table is checkpointed to S3.
3. The Machines exit and their volumes are released.

The instance moves through stopping to stopped. A stopped instance has no Machines and no volumes, so it costs storage only. Its URL stops answering; the LanceDB SDK gets a connection error, not a 503.

Stop when you want to pause an instance without losing data, or before a resize.

A stop is not the only time a table is checkpointed. A table that has gone five minutes without a query or an insert is checkpointed and closed on its own, and reopens on next use; see how many tables a tier keeps open.

Start

POST …/start, or Start, answers 202, creates Machines and volumes for the instance’s current shape and cache and moves the instance back to running once the instance can accept writes. The cache is rebuilt from S3 as queries arrive. The URL and the token do not change.

running means write-ready, so the first write after an instance reports running does not have to be retried. For a ramp instance that is its one Machine answering; for a launch cluster the control plane waits for the engine’s own /readyz to report that the cluster can write, because a node can answer as healthy while the cluster is still a member short of quorum. Waiting for a launch cluster to converge adds a few seconds to its boot.

A launch instance’s three nodes are created and launched together, not one after another. The first thing a node does is wait for a quorum of its peers, so nodes that come up together converge in seconds; starting them in sequence would make the first node wait out the creation of the last.

If the boot budget runs out, the instance goes failed and its error carries the engine’s own reason — for example instance provisioning failed: not ready: 1 of 2 required members are reachable; unreachable node-1, node-2 — rather than a bare timeout.

start also retries a failed instance.

Degraded

A launch instance keeps serving with one node down: its quorum is two of three. When the single-instance read (GET …/instances/:id, or the instance page) finds a running cluster that says it cannot write, the record carries a degraded object with the engine’s reason and the members it cannot reach:

{"state": "running",
 "degraded": {"reason": "1 of 2 required members are reachable",
              "unreachable": ["node-1", "node-2"]}}

The dashboard shows this as a banner on the instance page and marks the unreachable nodes in the Machines table. Reads keep being served; writes wait for quorum, which returns on its own once the members are back. A healthy cluster, a ramp instance and the instance list all report degraded: null — the list never asks the instances themselves.

Resize

POST …/resize with {shape?, cache?} changes the shape, the cache tier or both and answers 200 with the updated record. It is accepted only while the instance is stopped; otherwise the API answers 409 and the dashboard keeps the form disabled.

curl -X POST https://api.walleye.dev/v1/organizations/$ORG/instances/$INSTANCE/stop \
  -H "Authorization: Bearer $LAKEDAY_TOKEN"
# wait for "state": "stopped"
curl -X POST https://api.walleye.dev/v1/organizations/$ORG/instances/$INSTANCE/resize \
  -H "Authorization: Bearer $LAKEDAY_TOKEN" -H "Content-Type: application/json" \
  -d '{"shape": "launch", "cache": "medium"}'
curl -X POST https://api.walleye.dev/v1/organizations/$ORG/instances/$INSTANCE/start \
  -H "Authorization: Bearer $LAKEDAY_TOKEN"

Old volumes are deleted on resize. That is safe because the cache is disposable and every acknowledged write is already in S3, so a drained stop followed by a resized start loses nothing. Moving from ramp to launch starts three Machines from the same bucket.

The move is one way. Once a bucket has been served by a launch cluster it cannot go back to ramp: the Bitr log has already used write positions that a single node’s object-store log would reuse. A resize from launch to ramp answers 409 with {"error":"an instance cannot be resized from launch back to ramp"}, and the dashboard disables the option. Cache changes on a launch instance remain allowed.

Rotate the token

POST …/token, or Rotate token, mints a new instance token and returns {token, url} once, the same way creation returned the first token. The previous token stops working as soon as the Machines pick up the new secret. Rotate when a token has leaked or when a client is decommissioned.

A Machine only reads a new secret when Fly stages it into a new version, and that replaces the Machine. So the rotation rolls: one node is replaced, and only once it is started, the instance answers again and the cluster reports it can write is the next node touched. A launch instance therefore keeps serving throughout — at most one of its three nodes is ever out, and two is quorum. A ramp instance is one Machine, so it is briefly unavailable while that Machine comes back.

The cutover is not instant, because a node only reads the new secret once it has been replaced. The previous token stops working as soon as the first node comes back, and the new one is accepted by every node once the roll finishes — about one node restart per node, roughly twenty seconds for a three-node cluster. In between, which token a request is accepted with depends on which node it lands on, so a client should retry a 401 for the length of the roll rather than treat it as a bad token. Reads and writes with a token a node accepts are served throughout; nothing goes down.

A rotation interrupted partway is retried and rolls from the first node again. The instance stays running the whole time, and stop and delete are accepted throughout: a roll is how the instance converges on a token it already has, so neither waits for one. Stopping mid-rotation is safe — the new token is in the app’s secrets before the first node is touched, so the next start carries it. The retries are bounded: if a node keeps refusing, the rotation gives up and the instance’s error says that the rotation did not complete and which node it stopped at. Rotate again once the underlying problem is fixed.

Upgrades

Lakeday releases a new engine image from time to time. An instance picks one up when it is created or started, and a long-running instance is moved onto it by an upgrade. You do not schedule these; Lakeday rolls them across the fleet, a canary set first and then in waves, and stops the whole rollout on the first instance that does not come back.

What an upgrade costs you depends on the shape, and the two are genuinely different:

• A launch instance is upgraded node by node, exactly as a token rotation rolls. One of its three nodes is replaced at a time and the next is not touched until the engine reports the cluster can write again. Two of three is quorum, so writes are served throughout and nothing acknowledged is lost.
• A ramp instance is one Machine, so it restarts: SIGTERM drains and checkpoints, the new version boots on the same volume, and the instance is unavailable for the few seconds that takes. There is no second member to serve in the meantime, and we would rather say so than imply every tier is zero-downtime. A write in flight during the restart fails and should be retried; a write already acknowledged is checkpointed by the drain and is not lost.
• A ramp small instance that Fly has suspended is not woken. It is given the new version where it sits and boots on it at the next request, at no cost at all.
• A stopped instance is not touched. It comes up on the current image the next time you start it.

Delete

DELETE …/instances/:id, or Delete with the instance name typed to confirm, answers 202 and removes the Fly app, its Machines and volumes, and the S3 bucket with every table. The record stays readable by id in state deleted and leaves the list. This is not recoverable. Deleting an organization deletes all of its instances the same way.