Benchmarks

Iso-recall — latency at equal recall (CLIP-512, 25k)

The fair comparison: tune each system's search effort to the same recall, then compare latency (lower-left of a curve is better). LanceDB uses nprobes+refine_factor; DreamDB uses its query-time nprobe knob; Deep Lake has no query knob (single point).

At moderate recall LanceDB is a touch faster; at ~0.99 the two are neck-and-neck (LanceDB ~5.8 ms, DreamDB ~6–10 ms); DreamDB reaches marginally higher recall (0.997). Both are far below Deep Lake's fixed ~12 ms. At 25k all three are close — the separation sharpens at 1.7M (below).

At scale — 1.74M CLIP-512 (k=10)

25k is near-trivial; scale is where architectures separate. On the full 1.74M-vector openimages corpus, latency (p50) at each recall target. DreamDB is competitive with LanceDB and shown compacted (the operating state after a bulk load):

DreamDB recall 0.943 / 0.973 / 0.993 at np16 / np32 / np128. Deep Lake's binary path collapses to a brute-force scan at this size — recall 0.51 @ 1.74 s, failing every target. Storage ~3.6 GB all (DreamDB 4.2 GB with rerank; 7.9 GB transiently post-compact, pre-gc).

Read. At 1.74M, DreamDB is within ~1.2× of LanceDB at recall 0.90 and ~1.3× at 0.95, and is the only system to cleanly reach ≥0.99 (LanceDB's PQ plateaus ≈0.95; Deep Lake degrades to a linear scan). Latency scales gently with N (100k 2.9 ms → 800k 4.9 ms → 1.74M 6.58 ms at recall ≈0.94). A bulk load should be followed by compact() — lossless and recall-identical.

Compaction — bulk-load maintenance (1.74M)

A bulk ingest commits in batches, and each commit writes one bucket fragment per IVF cell — so a cell ends up split across ~80 fragments and every query pays to read them all. compact() (spec/0021) merges each cell's fragments back to one: read-online, lossless. Measured before/after on the 1.74M corpus (same dataset + queries; collapsed 96,872 fragments across 1,256 cells in 7.9 s):

Recall is bit-for-bit identical (lossless merge) — fragmentation is the dominant cost of a raw bulk load, so merging fragments back to one per cell is the key post-ingest step, and the compacted column is the operating state in the at-scale table above. Caveat: compaction is copy-on-write, so on-disk footprint temporarily ~doubles (4.2 → 7.9 GB) until a gc pass reclaims the old fragments — compact + gc is the full maintenance cycle.

Random access by key (point lookup)

The episodic-memory workload: fetch a record by its time anchor (not similarity search). DreamDB uses a primary-key (time-anchor) index; LanceDB a columnar take. p50 by corpus size:

Flat (O(log N), N-independent) — ~0.07–0.10 ms across the full 20k→1.74M range, on par with (or ahead of) a columnar take. The PK index (`build_anchor_index` → `get_by_anchor`: root → leaf → one ranged record GET) replaces the naive temporal scan, whose fan-out grew ~√N (5.5 ms @200k → 17 ms @800k → 35 ms @1.74M → seconds at 10B). A primary key now has a primary index.

Capability matrix

✓ native · ~ partial · ✕ no. Click any capability for an explanation. Unique to DreamDB vs both LanceDB and Deep Lake: the top five rows.

CLIP-512 — DreamDB's target (25k, k=10)

High-dim normalized embeddings — DreamDB's design point. All three reach ~0.99 recall; differences are latency + the tunable curve above.

SIFT1M — low-dim L2 (1M×128, k=10)

Adversarial for a cosine + 1-bit design. 1-bit DreamDB is weak here (0.21); 4-bit reaches 0.80 — ahead of LanceDB (0.54) and Deep Lake (0.43) — trading ~1.35× storage and higher latency. SIFT was a config (bit-depth), not a fundamental loss.

Honest takeaways

• Retrieval: competitive on embeddings once tuned; LanceDB usually a touch faster at equal recall, both beat Deep Lake on latency.
• Competitive at scale: at 1.74M DreamDB is ~1.2× LanceDB at 0.90 recall (6.58 vs 5.56 ms), uniquely still reaches ≥0.99, and Deep Lake collapses to a brute-force scan. A bulk load wants a compact (+gc) pass — lossless and recall-identical.
• Remaining costs: slower ingest (per-batch commits), a gc pass after compact, and no high-QPS serving layer (out of scope by design).
• The wedge is the model: versioned + temporal + multimodal + multi-modal retrieval + serverless/in-browser — the intersection none of the others offer together.
• Not a tabular analytics warehouse (LanceDB), a training-loader lake (Deep Lake), or an enterprise versioning layer (lakeFS).