Build1 publisher3 min readPublished
Moving rekuiper's metadata catalog into RAM took the disk off its 100k msg/s hot path
The Rust reimplementation of eKuiper now hydrates streams, rules and auth keys from SQLite at startup and serves them from a lock-guarded HashMap, after profiling blamed flash I/O on eMMC gateways for the engine's burst-load packet loss.
The Engineer · Build desk
What happened
- Earlier rekuiper benchmarks held 100,000 messages per second on a single pinned core, and at an offered 200,000 msg/s the engine dropped packets or backlogged upstream.
- Profiling under burst loads put the bottleneck at the disk, not stream parsing or window math, with SQLite metadata reads stalling the Tokio runtime thread on slow flash.
- Internal actor queue depths grew from 1,024 records to 32,768 to absorb operating system scheduling jitter before backpressure reaches the MQTT network loop.
- The team then searched each workload's ceiling at 1,000 msg/s resolution against a strictly bounded Mosquitto broker instead of testing rounded rates.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A bench rig on NVMe cannot reproduce this ceiling, because the lookups that stall the engine cost microseconds there; the fault needs eMMC or microSD to show up at all.
- exposure Rule edits now return from memory and reach SQLite later, so a gateway that loses power after an API call can restart without the change the API confirmed.
- cost Headroom is bought in RAM at 32 times the previous per-actor queue depth, on a project whose earlier pitch was a 5 to 10 MB memory bound measured before the catalog moved into memory.
Ten milliseconds of I/O wait at 150,000 msg/s ingress leaves 1,500 messages in the socket buffer, and under QoS 0 Mosquitto drops them [8]. 150,000 times 0.010 seconds is 1,500 messages [24].
The old actor queues were 1,024 records deep [16]. At 150,000 msg/s that is 6.8 milliseconds of headroom [20], less than the stall the profile describes. The buffer drained into backpressure before the flash write returned. At 32,768 records, the same rate buys about 218 milliseconds [21].
The catalog change is the part I would copy. `MemoryCatalog` keeps streams, tables and rules behind `parking_lot::RwLock<HashMap<...>>` in `crates/rekuiper-core/src/catalog.rs` [9]. The daemon hydrates all of it from SQLite once, at startup [10]. After that the stream bus, rule evaluators and REST query endpoints read straight from the map, which the project describes as nanosecond lookups with zero system calls and zero disk I/O [11]. Writes run the other direction: memory first, SQLite asynchronously in the background [12]. One RwLock per map also means a rule edit takes a writer lock that every hot-path reader waits behind, so the design holds while operator edits stay rare next to lookups [9].
For the headline rate to mean anything on your hardware, a few things have to be true. The storage has to be the kind that stalls: the write-up puts SQLite lookups at microseconds on a developer NVMe drive, and blames I/O wait spikes on slow eMMC flash and microSD [6]. The offered load has to sit above the band where nobody notices, which the same post puts at 5,000 to 20,000 msg/s [7]. And the loss mode has to be the one measured, a dropped QoS 0 packet at a bounded Mosquitto broker [8][17].
The auth path gets the same treatment. Public RSA keys for JWT signature verification, loaded from `KUIPER_AUTH_PUBLIC_KEY_FILE`, are parsed and cached in memory, so an ingest request no longer reads a key off disk per request [13]. Config overlays under `/etc`, source definitions and JSON descriptors are cached on first access [14]. SQL sinks now share connection pools, and relational sinks emit parameterized multi-row `INSERT` chunks instead of one query per record [15].
The cost shows up in memory. The earlier benchmark's selling point was 5 to 10 MB across five MQTT workloads, against Go-based engines that climbed to hundreds of megabytes or failed outright [2]. Those figures predate a 32x increase in queue depth [22] and a catalog that is now resident in RAM [10]. The post does not include a new footprint.
On the measurement, the project is candid about what it did not know. "100k passed and 200k failed" left 100,000 msg/s unexamined [3], and a search at 1,000 msg/s resolution implies up to 100 candidate rates per workload [23]. The section describing the rebuild stops before the per-workload results; the up-to-200k figure appears in the title [19][25].
What to watch
- Whether the published per-workload ceilings land near 105k or near 195k msg/s when the full v0.500 results appear.
- Whether the asynchronous SQLite commit path gains a synchronous option for sites that cannot lose an operator's rule edit to a power cut.
- A measured memory footprint for 32,768-deep actor queues on a device previously sold on a 5 to 10 MB bound.