Benchmarks: the full method and every number

HoloDb is an embeddable, transactional SQL database for .NET that stores data as holograms in a single associative store. One realistic workload, a billing SaaS's invoice ledger, run on five engines (SQLite, DuckDB, SQL Server, HoloDb embedded and HoloDb server), then the optional Pleroma plane, single-row writes and transactions. Every figure is one run on one machine and carries its experiment id. Every result is verified identical across the engines before it is timed, including the queries where HoloDb trails, and every loss is shown.

How every figure was measured

• One run, one machine. Every figure on this page is one run on one machine (16 cores, AC power, idle, no other benchmark or heavy application running; the runs are dated 12 October 2026) and carries its experiment id. The plane showcase below is the exception on conditions: one run on 13 October 2026, not on an idle machine, with its conditions stated there.
• Written down first. Outcome thresholds and noise tests were written down before each run, every timed query first had to return identical rows from SQLite and from HoloDb (the billing workload from all five engines), and a cell whose own two timing loops disagreed by more than 25% is marked as not interpreted.
• SQLite first. SQLite (in-memory, the external reference) is listed first in every table. In the plane benchmark the HoloDb engine is the same database without a plane (its own maintained totals, zone maps and block-wise column scans) and the HoloDb plane is a database with the optional Pleroma plane attached.
• Medians. Times are medians of repeated calls. "Fresh process" means the first statements of a newly started process, which is what an application feels after it starts; the cost of just-in-time compilation is included.

Billing workload: HoloDb against SQLite, DuckDB and SQL Server (EXP-PB12, 10,000,000 invoices)

The workload. A billing SaaS's invoice ledger, the shape of data a subscription business runs its dashboards on: a normalised invoices fact joined to a customers dimension.

The method. The same data (fixed seed) and the same SQL on five engines, each loaded through its own bulk path: SQLite (in-memory), DuckDB (in-process, in memory, its fastest mode), SQL Server 2022 LocalDB (on disk, in tempdb), HoloDb embedded (in-process) and HoloDb server (networked over TLS on the loopback interface, the fair peer to SQL Server's client/server round trip). Sixteen queries a billing business actually runs: five rollups (revenue by status, month and region, the total, the count), a single-invoice lookup, five single-table scans (overdue outstanding, big-ticket count, largest invoices, top customers, a region and status breakdown) and five cross-table joins to the customer dimension (revenue by segment, by segment and status, enterprise revenue, by signup cohort, top named customers). Before any query is timed, its full result set is normalised and compared across all five engines; every one of the 16 results was identical. Times are the median of 5 runs of execute plus consume-every-row through each engine's typed client API, on a 16-core machine on 12 October 2026, started at 10 percent CPU with 17.4 GB of memory free and no other benchmark or heavy application running. It is one run on one machine, and it is the final run of this comparison: it supersedes the two earlier runs of the same benchmark on the same machine the same day (EXP-PB7 and EXP-PB9), against which it agrees on every engine's unchanged cells (the DuckDB and SQL Server medians are about 7 percent higher in this run). Bulk load, rows per second: DuckDB 1,159,279; HoloDb server 1,051,473; SQLite 726,050; SQL Server 693,231; HoloDb embedded 679,271.

In-process, in memory: HoloDb embedded against DuckDB (and SQLite), median ms

The purple cell is HoloDb where it is faster than DuckDB; the green cell and the amber note mark the rows where another engine is ahead: DuckDB on the signup-cohort join, and SQLite on the single-invoice lookup (0.01 ms against HoloDb's 0.02 ms). Ties carry no colour.

HoloDb embedded is faster than SQLite on 15 of the 16 queries, from 48x (top customers by revenue) to more than 2,700x on the rollups, and SQLite is ahead on the single-invoice lookup (0.01 ms against 0.02 ms at the benchmark's two-decimal resolution). Against DuckDB, counting a difference under 1.15x as a tie, HoloDb is faster on 13 queries, tied on 2 and slower on 1. The five rollups and the lookup read a maintained total or the primary-key index, so they take microseconds and do not grow with the ledger (the benchmark prints two decimals, so these cells are shown at that resolution and the ratios are lower bounds). Where HoloDb lost: the signup-cohort join, 1.25x behind DuckDB (a parallel scan of the 10 million invoices at about 80 ns per row per core against DuckDB's columnar engine), and the single-row lookup against SQLite.

Networked: HoloDb server against SQL Server 2022 LocalDB (and SQLite), median ms

The purple cell is HoloDb server where it is faster than SQL Server; the green SQL Server cell and the amber note mark the one row where SQL Server is faster.

Both are networked services paying a client/server round trip; HoloDb serves its working set from RAM and SQL Server reads its on-disk table. HoloDb server is faster than SQL Server on 15 of the 16 queries (including all five joins, by 21x to 185x), ties on none and is slower on the single-row lookup, where SQL Server answers in 0.07 ms and HoloDb server in 0.18 ms (SQLite, in the same process as its caller, takes 0.01 ms). The server figures are the median of five calls each, so they include the first-call cost of the network path: a separate longer loop through the same client measured the rollups and the lookup at a steady 0.06 to 0.10 ms (a point query pays a fixed 0.07 ms of client, TLS, socket and result encoding; the same query takes 1 to 3 us embedded), and five-call medians in fresh processes read 0.17 to 0.26 ms (median over eight fresh processes per query; single processes 0.11 to 0.34 ms).

Highlights from this run

• Total invoiced across 10,000,000 invoices (SELECT SUM(amount_cents)), same query, same data, all results identical: SQLite 414.54 ms, DuckDB 2.56 ms, SQL Server 943.63 ms, HoloDb under 0.01 ms (a maintained total is read, not 10 million rows).
• The five rollups and the single-invoice lookup resolve in microseconds in the embedded engine, and in 0.17 to 0.35 ms as a network service.
• As a networked service HoloDb is faster than SQL Server on 15 of 16 billing queries, including every join.
• On the five joins the embedded engine is faster than DuckDB on revenue by segment (3.3x), top named customers (2.2x) and segment and status (2.1x), ties it on enterprise revenue, and is 1.25x behind on the signup-cohort join.

Release check (EXP-PB13)

Before 5.1.0 shipped, every benchmark cell was run twice on the release build, back to back on an idle 16-core machine: the 10-million-invoice comparison, the plane benchmark at 100,000, 300,000 and 1,000,000 rows, single-row writes and transactions, and opening a 1,000,000-row database from its log and from a checkpoint. The join count on a 100,000-row table joined to a 40-row table takes about 0.07 ms.

Opening a database

Opening a database of 1,000,000 rows (a 43 MB log) takes about 2.0 s by replaying its write-ahead log (1.95 to 2.05 s across two runs) and about 1.1 s from a checkpoint (1.09 to 1.11 s), with the loaded state checked correct and a heap of about 110 bytes per row. Conditions: in process, a newly started process each time, an idle 16-core machine on 12 October 2026. This is the engine's own recovery and checkpoint; it does not use the plane or the page store.

The optional Pleroma plane on the billing workload (EXP-PB11)

The plane is not on in the comparison above: it is opt-in per table (EnablePleroma with explicit options). This section is a separate run of the same workload (10,000,000 invoices, all 16 results again identical across all engines) with a plane attached to the invoices table: dimensions region (4 values), status (5) and month (24), a 10-bit joint, and the measure amount_cents. It takes 34,528 bytes of plane cells (+0.3 MB of heap) and 741 ms to build for 10 million rows (749 ms on the server's database). The same database is timed with the plane off and with the plane on (default routing), so every other column of the run is unchanged.

Through the server the same two queries take 6.30 ms and 34.58 ms with the plane off and 0.11 ms and 0.45 ms with it on (SQL Server: 611.59 ms and 2,898.13 ms). What the plane serves: filters and multi-column GROUP BY over its dimension columns. What it declines, and why: the big-ticket count (the filter is on a measure, not a dimension), top customers (200,000 distinct customer ids, far over the 1,024-value dimension cap), the five joins (200,000 join keys against a limit of 4,095, and no plane on the customers table), and the lookup and largest-invoices queries (no aggregate). The five rollups are left to the engine on purpose: it reads a maintained total there in microseconds, and the plane is slower. Per database, 12 calls were plane-served (two queries, six calls each) and 42 declined (seven queries, six calls each), exactly the split above. Noise: the cells the plane does not touch run the identical engine path in both columns, and they differed by 1 to 13 percent typically and by up to 33 percent (big-ticket: 3.01 ms off, 2.26 ms on), so differences of that size are noise. The server rollups read 0.12 to 0.32 ms off and 0.14 to 0.17 ms on, which is consistent with the first-timed column of each query paying the cold first calls of the network path; the plane does not serve these queries.

The plane's SQL form (CREATE INDEX ... USING SUMMARY, SHOW INDEXES, EXPLAIN) and the Pleroma page store are described, with their limits, in Built on Pleroma.

Where the plane is the right tool (EXP-PS1)

The optional Pleroma aggregate plane (described under "Built on Pleroma" below) pays off on three kinds of workload, and it has costs that sit next to the wins. These workloads were constructed to suit the plane and are measured separately from the engine-only comparisons below, which do not use it. These figures are the EXP-PS1 plane showcase, one run on one machine on 13 October 2026 on the build released as 5.1.1: an 11th Gen Intel Core i9-11950H at 2.60 GHz (16 logical cores), 18.1 GB of 31.7 GB memory free at the start, no other .NET process running, with ordinary desktop applications and the SQL Server service open, so this is not an idle-machine run. Every HoloDb figure is an in-memory database with no write-ahead log; SQLite and DuckDB ran in memory and SQL Server LocalDB used tempdb on disk. Each cell is the median of the lower of two timing loops; the per-round figures in B are medians over rounds. Every timed query returned identical results on all engines, and after each workload PleromaVerify found the plane equal to a rebuild from the columns. "Engine" is HoloDb without a plane, "plane" is HoloDb with the plane attached and default routing, both embedded.

What the three workloads have in common: the filter combinations cannot be pre-indexed because nobody knows in advance which dimensions a person will cut by, the answers must be exact, and the table keeps changing while it is queried.

A. A live dashboard: 10,000,000 events, 32 ad-hoc slices

Six small-domain dimensions (a 14-bit joint) and the measures. Building the plane for 10,000,000 rows took 1,759 ms and its cells take 920,248 bytes. The 32 slices are scalar filters (W), group-bys with and without a filter (G), HAVING (H), COUNT DISTINCT and MIN/MAX with a filter (D), plus queries outside the plane (X) and whole-table queries the engine already answers from a maintained total (S).

The best other engine on the W, G, H and D slices is DuckDB in every row shown. In W, three of the nine cells (A10, A11, A12) are not interpreted because the engine's own two timing loops disagreed by more than 25%; the class median is as printed.

First calls. The first call of each query, which pays any one-time work, is also faster with the plane: the median first-call speed-up over the engine is 16.99x for W, 10.90x for G, 13.97x for H and 10.17x for D. A04 takes 1.047 ms on its first call with the plane against 17.792 ms for the engine. Across the first four rows below, the plane takes 0.153 to 2.110 ms, the engine 18.526 to 50.189 ms and DuckDB 25.185 to 67.130 ms.

The queries the plane declines or leaves to the engine (the X class: a filter on a non-dimension column, COUNT(DISTINCT user_id), a sum of a non-measure column, a top-10 by user, a filter on a measure, a row list and a point lookup by event id) run the engine's own path with the plane attached and tie with the engine: the seven X queries read 0.95x to 1.04x warm. The plane arm is ahead of the best other engine on them by 1.1x to 3.3x; A25, A28 and A31 are ahead by only 1.1x, and the 3.3x is the point lookup (0.002 ms against SQLite's 0.007 ms). On the first call, five of the seven (A25, A27, A28, A30, A31) were slower with the plane attached, at 0.42x to 0.84x of the engine's first call; these are single samples, and the warm ratios are 0.95x to 1.04x.

B. Live ingest while querying

A 2,000,000-row base table, then 20 rounds of a 10,000-row bulk load, 200 single-row INSERTs and an eight-query dashboard panel, then 300 trickle pairs (one INSERT followed by one group-by query). The panel's results matched in every round. Per round, medians:

The plane makes the panel 11.95x faster than the engine and the round total 5.78x faster; the trickle pair is 7.00x faster. Through the server the round total is 179.34 ms for the engine and 45.65 ms with the plane. What the plane costs on the write side: the 10,000-row bulk load takes 11.62 ms against 9.85 ms (1.18x), and 200 single-row INSERTs take 1.55 ms against 1.23 ms (1.26x), about 7.8 microseconds against 6.2 per INSERT. The first panel query after each ingest (P1, the A04 shape) gains only 1.66x (2.304 ms against 3.835 ms), while the other seven gain 2.87x to 319.60x. The likely cause is the flush of the plane's pending changes that the first query after an ingest triggers; the flush was not timed separately. DuckDB takes 6.288 ms on that query, so the plane is still ahead of it (2.7x), and on the others the plane takes 0.202 to 2.082 ms against DuckDB's 5.147 to 28.316 ms.

C. Multi-tenant: slices and joins

Five million events spread over six tenant tables (2,000,000 down to 150,000 rows) and a 2,000-product dimension table.

Tenant slices. A filter and a COUNT and SUM on one tenant (C0c to C5c): median 11.70x faster than the engine and 25.0x faster than the best other engine. The largest tenant reads 0.171 ms with the plane, 4.263 ms for the engine, 6.312 ms for DuckDB and 195.195 ms for SQL Server. The C2c cell is not interpreted (its two timing loops on the HoloDb arms disagreed by more than 25%); the medians are as printed.

Joins, stated exactly. The join in these tests is the same statement repeated: tenant events joined to the products table on the product key (about 2,000 keys), with COUNT(*) alone (a) or COUNT(*) and two SUMs (b). The plane serves a repeat from a memo of the exact answer, which by design (and by the joincache checks, not by this run) is valid until either of the two tables is written. So the very low figures below are for a repeated identical statement with no write in between:

Embedded, a memo hit takes 0.001 to 0.003 ms; through the server it takes 0.071 to 0.084 ms, against DuckDB's 0.947 to 2.906 ms and the engine's 0.110 to 2.716 ms for the same joins. A join that is not a memo hit usually costs more than the engine's own join. The first call of a statement costs more through the plane than through the engine in 11 of the 12 join cells. A call after a write to either table was not timed in this run; by design it runs the same computation as a first call, but a first call also pays one-time costs, so treat the first-call figures below as indicative of it, not as a measurement. First calls in this run: join COUNT 1.016 to 2.921 ms through the plane on the five later tenants (1.372, 1.088, 2.921, 1.016, 1.245 ms) against 0.179 to 0.573 ms for the engine; the first join of the whole run, which also pays one-time start-up, was the one cell where the plane was lower: 13.800 ms against the engine's 18.540 ms. Join COUNT and two SUMs 2.348 to 8.134 ms through the plane (8.134, 6.230, 3.138, 2.610, 4.498, 2.348 ms) against 1.364 to 3.076 ms for the engine. If the tables are written between nearly every read, expect the plane join to cost more than the engine's own join, and the plane's value on such a workload to come from the filters and group-bys above, not the joins. Joins with a WHERE or a GROUP BY on the dimension table are not served by the plane at all; they run the engine's path and tie with it (C-X1 to C-X4, 0.93x to 1.02x).

The checks that guard these figures

Queries the plane declines or leaves to the engine moved within noise: of the 16 queries that run the same path in both arms, none fell outside the registered band of 0.8x to 1.25x, and the class medians were 0.97 (workload A) and 0.99 (workload C). A placebo built to catch a wrong plane (the measure column shuffled at 1,000,000 rows) was caught on every query it was registered to change and matched on every query it was registered not to change: 0 failures of 32. PleromaVerify found the plane identical to a rebuild from the columns after each workload.

Exactness (Pleroma plane and page store)

Proven by checked-in verbs in HoloDb.Bench (pleromaparity, pleromastore, and the engine suites listed under each result).

• Exact, not approximate. The aggregate plane returns row-for-row the same answers as the engine and SQLite across the parity suite (2,853 checks at 2,000 rows; the suite also passes at 5,000 and 100,000 rows), including a random insert, update, delete, rollback and reopen log, and sums past 64 bits. Nine deliberate mutations (a dropped carry, a missed delete, a stale join limb and others) each turned the suite red.
• Sums past 64 bits are never wrapped. The plane raises an error and PleromaSumExact returns the exact 128-bit total.
• The page store survives faults. 72 of 72 store checks pass: exactly-once replay after a crash, torn journal tail discarded, restore points that rewind without recounting, and repair of a lost page from its parity group. Eight deliberate mutations and a tiny-cache case each turned the suite red.
• Plane memory is flat. The cells of a 14-bit joint plane are about 0.92 MB whatever the row count (920,248 bytes at 10,000,000 rows, 922,936 at 100,000 to 1,000,000), plus the same again in pending deltas before a flush.

Single-row writes and transactions, engine alone against SQLite (EXP-PB4, EXP-PB5)

Autocommit statements on a table of the benchmark shape (9 columns), 205 statements per cell, the first 5 dropped, median. EXP-PB5 part C, fresh process (each size in its own newly started process):

Transactions (EXP-PB4, 1,000,000 rows; the 100,000-row figures follow the same pattern; this experiment ran before the compilation setting of EXP-PB5 was added, so these engine times are an upper bound and were not re-measured):

BEGIN costs the same at any table size because rows are journaled as they change instead of the table being copied; ROLLBACK replays the journal. Atomicity and crash recovery are covered by acidstmt (48 checks: a failing log append is undone in full, a reopen from the log reproduces the committed state) and txnundo (58 checks plus about 600 randomized comparisons against SQLite; three deliberate mutations each turned it red).

Queries without the plane, and the plane, at 1,000,000 rows (EXP-PB2 re-run, EXP-PB3, EXP-PB5)

Seventeen query classes on a 1,000,000-row table. This is the re-run of the pre-registered benchmark after the engine's scan, group-by, join and transaction work; its control query (a query the plane declines, so both sides run the same engine path) read 0.81 against a registered band of 0.8 to 1.25, so the measurement is unbiased, narrowly. An earlier pass of the same size read 0.73 and is not reported.

The join rows use a very small dimension table. The benchmark joins the 1,000,000-row table to a 40-row table: the two INT-key joins have 40 distinct join keys, and the TEXT-key join has 4 distinct keys (4 labels, each on 10 of the 40 rows). Those key counts are far below the roughly 2,000 product keys of workload C above and the 200,000 join keys of the billing workload, so these join figures say little about joins over many keys.

How to read it. The engine alone is faster than SQLite on every class, from 9x (the TEXT join) to more than 250x on COUNT(*) and more than 20,000x on SUM, single-column GROUP BY, MIN/MAX and COUNT DISTINCT over the whole table. The plane is faster than SQLite on every class it serves, and against the engine it now wins where there is a real scan or join to avoid: the filters, the joins, COUNT DISTINCT with a WHERE. It does not win the whole-table and single-column classes, where the engine reads a maintained total in nanoseconds (the default routing, PleromaMode.Auto, sends those to the engine), and it only ties the two-column GROUP BY at this size. Outcome counts for the plane against the engine: 7 classes at least 2x faster, 3 ties, 1 between 1.25x and 2x, 4 slower, 2 not interpreted.

Other sizes of the same re-run (the control read 0.99 at 100,000 rows and 1.11 at 300,000, both inside the band). At 100,000 rows the engine is faster than the plane on seven of the 17 classes: the whole-table classes, the single-column GROUP BY and MIN/MAX (0.33x to 0.51x), the two-column GROUP BY (0.26x), GROUP BY with a WHERE (0.41x) and COUNT DISTINCT with a WHERE (0.58x); the plane is faster on the three-column WHERE (5.2x), LIKE ... OR (15x), BETWEEN (29x), the range count (62x), the equality filter (95x), the join SUM (41x) and the TEXT join (2,700x). At 300,000 rows the plane is faster on every filter and join (24x to 1,887x) and on the two-column GROUP BY (1.77x), and the engine is faster on GROUP BY with a WHERE (0.73x). Default routing sends GROUP BY with a WHERE to the engine below 140,000 rows.

Costs of the plane when it is attached (same re-run, plane against the engine alone)

• Single-row INSERT: 15.4, 9.3 and 16.1 us against 8.3, 6.5 and 9.6 us (100,000, 300,000 and 1,000,000 rows), so the plane adds 1.4x to 1.9x. UPDATE: 24.2, 13.3 and 27.4 us against 13.6, 8.3 and 15.8 us (1.6x to 1.8x). These cells are microseconds and move with the order the two arms run in (the INSERT ratio at 100,000 rows read 1.9x or 2.3x depending on which arm ran first).
• Batch load of the whole table: 1.93 s with the plane against 1.80 s without at 1,000,000 rows (1.07x); SQLite loaded the same rows in 1.80 s. Rebuilding the plane from the columns takes 131 ms at that size.
• One INSERT followed by a two-column GROUP BY, repeated: engine 1.35 ms, 4.25 ms and 8.16 ms per pair against the plane's 3.24 ms, 2.89 ms and 3.84 ms (100,000, 300,000 and 1,000,000 rows), so the plane pays a roughly 3 ms floor that the engine is faster than on small tables and slower than on large ones.
• Memory: the plane's cells are 0.92 B per row at 1,000,000 rows (a flat 0.92 MB joint slice whatever the row count); the database's heap is 139.7 B per row with the plane against 135.2 B without, a difference of about 4.5 MB (1.9 MB in the earlier 1,000,000-row runs, 1.9 MB at 300,000 rows and 4.6 MB at 100,000 rows).
• After all the writes the plane and the engine agreed on all 17 queries and the plane equalled a rebuild from the columns.

Single-row DELETE inside this benchmark is not reported: its loop draws ids with replacement and mixes in transaction overhead, so it does not measure the statement. The single-row DELETE figure is the autocommit one in the table above.

The TEXT-key join is faster than SQLite in the engine alone (EXP-PB3: 39.98 ms against 322.17 ms at 1,000,000 rows, rows identical). It requires both join columns to be TEXT; a join between an INT column and a TEXT column takes the general path and is much slower.

Limits that are still open

One run per size on one machine; one synthetic table (9 columns, four small-domain dimensions, a 14-bit joint). The plane serves only the columns you attach, lives in RAM and is not persisted. In the engine alone: GROUP BY with a WHERE is 3 ms at 1,000,000 rows because the filter and the grouping are separate passes; the dense-code GROUP BY covers INT and TEXT columns whose combined distinct count is at most 65,536, and anything else (REAL group keys, higher cardinality) uses a slower string-key path; REAL aggregates over a filter use the older scan; a 1,000-statement transaction is 4 to 5 times SQLite's cost; after a ROLLBACK of deleted rows the table holds the same rows but not necessarily in the same internal order. The first DDL statement inside a transaction takes a snapshot of every in-RAM table (so it costs time proportional to the data), and autocommit CREATE INDEX and ALTER clone the one table they change. The server single-row lookup includes about 0.07 ms of client, TLS and socket cost that is outside the engine; SQL Server's 0.07 ms and SQLite's 0.01 ms are both below HoloDb server's 0.18 ms, and SQLite is ahead of embedded HoloDb on the lookup (0.01 ms against 0.02 ms). Still behind DuckDB on the billing workload: the signup-cohort join (1.25x, a parallel scan of the 10 million invoices at about 80 ns per row per core); tied with it on the big-ticket count and the enterprise-revenue join. The Pleroma plane is not part of the main comparison and does not help the billing joins (200,000 join keys against the limit of 4,095) or the top-customers query (see the plane section above); a plane join over fewer keys is served, as in workload C above. A GROUP BY with more than 64 distinct values is not served from the live snapshot and can wait behind a continuous writer (see the dashboard-read measurement). Page-store throughput is not reported here: earlier figures came from other conditions, were not published, and are not comparable with a current run.

The aggregate plane is separate from the 5.0.0 page-store additions (check prime, parity, striped reads): none of them is on the plane's path, and the benchmarks above do not exercise them.

← Back to HoloDb  ·  Run it in the browser →