see in the dark

An open-source LanceDB server. Every write is durable in object storage, and reads come from memory and disk.

One binary. Your bucket.

Worked example

Parse a stream straight into a table.

Read records off a socket, a log tail or a queue, batch them, and write them into a Walleye table with the stock LanceDB client. There is no Walleye SDK to learn and no proxy in front of your data: the client speaks the LanceDB remote protocol straight to your server.

Python

pip install lancedb pyarrow
import json, os, sys
import lancedb, pyarrow as pa

db = lancedb.connect(
    "db://walleye",
    api_key=os.environ["WALLEYE_TOKEN"],
    host_override=os.environ["WALLEYE_URL"],  # where your server listens
    region="us-east-1",
)

schema = pa.schema([
    pa.field("id", pa.int64()),
    pa.field("body", pa.string()),
    pa.field("embedding", pa.list_(pa.float32(), 768)),
])
table = (
    db.open_table("events")
    if "events" in db.list_tables().tables
    else db.create_table("events", schema=schema)
)

rows = []
for line in sys.stdin:               # one JSON record per line
    event = json.loads(line)
    rows.append({
        "id": event["id"],
        "body": event["body"],
        "embedding": embed(event["body"]),
    })
    if len(rows) < 500:
        continue
    # Returns once the batch is durable in object storage.
    table.add(pa.Table.from_pylist(rows, schema=schema))
    rows = []

if rows:
    table.add(pa.Table.from_pylist(rows, schema=schema))

print(table.count_rows())
print(table.search(embed("checkout failed")).limit(5).to_list())

JavaScript

npm install @lancedb/lancedb
import * as lancedb from "@lancedb/lancedb";
import { createInterface } from "node:readline";

const db = await lancedb.connect({
  uri: "db://walleye",
  apiKey: process.env.WALLEYE_TOKEN,
  hostOverride: process.env.WALLEYE_URL, // where your server listens
  region: "us-east-1",
});

const names = await db.tableNames();
let table = names.includes("events") ? await db.openTable("events") : null;
let rows = [];

async function flush() {
  if (rows.length === 0) return;
  // Resolves once the batch is durable in object storage.
  if (table) await table.add(rows);
  else table = await db.createTable("events", rows);
  rows = [];
}

for await (const line of createInterface({ input: process.stdin })) {
  const event = JSON.parse(line);
  rows.push({
    id: event.id,
    body: event.body,
    embedding: await embed(event.body),
  });
  if (rows.length === 500) await flush();
}
await flush();

const query = table.search(await embed("checkout failed"));
console.log(await table.countRows());
console.log(await query.limit(5).toArray());

This is where durability earns its place. add does not return until the batch is durable in the instance's own object storage, so a crash mid-stream, whether the writer's, the instance's or the network's, loses nothing it acknowledged. The instance comes back with everything it acknowledged, and the count you printed is the count that is still there. Each writer and reader can hold its own scoped token, and revoking one restarts nothing.

Run it yourself

One binary and a bucket.

Build walleye-node, point it at a bucket, and it serves the LanceDB remote protocol. Add two more nodes behind the same address when one is not enough.

cargo build --release -p walleye-node

export WALLEYE_BUCKET=my-bucket
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export WALLEYE_TOKEN=$(openssl rand -hex 24)

./target/release/walleye-node

How it works

Durable in object storage.
Served from memory and disk.

  1. Every write is durable before it is acknowledged.

    A write is acknowledged only once it is durable. One node commits each write to object storage before it acknowledges it; three nodes replicate it across a quorum of two first and reach object storage within a second.

  2. Reads come from memory and disk.

    Hot fragments and indexes live in a thin cache on the node. Tables can be larger than the cache; colder data is read from object storage on demand. The cache is disposable and is rebuilt from object storage whenever a node is replaced, so losing a node loses no acknowledged write.

  3. One node or three.

    Start with one node and a bucket. Add two more behind the same address when you need them: every node serves the whole LanceDB API, and each table is owned by one of the three, so losing a node costs no acknowledged write, and costs service to that node's tables until it is back.

Read about durability

Managed hosting

Rather not run it?

Lakeday runs Walleye instances for you. Pick a size, and connect to it at db.lakeday.ai with a token. Your first month is free.

Managed hosting by Lakeday