Product1 publisher2 min readPublished
Ten thousand Meshtastic nodes produced 550,000 Prometheus series in 20 minutes
Gleb Tcivie's open source monitoring tool outgrew Prometheus once node identity in metric labels blew past the 10-second scrape budget. It now runs on one Postgres database with TimescaleDB.
The Product Desk · Product desk
What happened
- Meshtastic Metrics Exporter, an open source monitoring tool for mesh radio networks, ran on Prometheus, which held at low scale and stopped working once the networks grew past a few thousand nodes.
- Around 10,000 nodes appeared within 20 minutes of listening to the public Meshtastic MQTT server, and at roughly 55 metrics per node that is close to half a million metric series.
- Node identity such as name, hardware model and role has to sit in metric labels, and every distinct combination of those labels becomes another series.
- Node metadata already lived in its own Postgres database, so matching a node's identity to its history meant querying two stores and joining the results by hand.
- The exporter now writes node data and telemetry history into one Postgres database running TimescaleDB, and ships as three containers a mesh operator starts with a single docker compose command.
Compiled by The Product DeskSomething wrong?How this is made
Why it matters
- constraint The ceiling here is a 10-second deadline, so a bigger metrics box does not help; what has to come down is the count of label combinations.
- decision Teams shortlisting a time-series store for a high-churn fleet have to decide where entity identity lives before comparing ingest benchmarks, because the fastest ingester can still leave two databases to join by hand.
- cost The maintenance bill lands on one person with a day job. Partition management and cleanup jobs are the expensive line.
- capability Triage moves to whoever has the dashboard: an operator who writes no queries can open a node's full history from its id.
Prometheus gives a scrape job 10 seconds to finish, and the exporter was well past that budget [6]. Multiply out and one scrape had to collect 550,000 series [1], which is 55,000 series a second [2]. The roster was moving while he watched, too: about 500 new nodes a minute across the 20-minute test [3].
Swapping the metrics store would have fixed half of it. Tcivie looked at VictoriaMetrics and InfluxDB first and ruled both out, because each solved the series problem and left node metadata sitting in a different database [9]. TimescaleDB runs as an extension inside Postgres, so adopting it removed a database [8]. The Postgres instance holding metadata was already there [7].
"The ability to quickly insert data and not worry about size or partitioning, and the ability to easily query with natural SQL language, made it a really good choice that fit this project like a glove," Tcivie said [11]. In practice that is ordinary SQL joining node details to their history, with retention and compression declared once instead of cleanup jobs someone has to write and keep working [10]. He builds and maintains the project alone, alongside his work as a software engineer [16].
Tiger Data published the account in its Community Member Spotlight series, in which the company invites customers to describe their own work [18]. That 10,000-node figure comes from a test against the public Meshtastic MQTT server, not a census of a production network [3]. The stack is specific enough to check: a Python service subscribes to the mesh MQTT feed, decodes the Meshtastic protobufs, validates each device and writes the measurements into Postgres 16 [12]. Other community-run networks have picked it up through the global Meshtastic Discord [17].
For any fleet with a churning roster, it comes down to whether the attributes that identify a thing have to travel with every reading it emits, and whether a routine support question requires joining those attributes to the history. Both are answerable before a migration starts. When both answers are yes, the figure to compute is entities times metrics per entity divided by the scrape window you actually have, and when that exceeds what one scrape can move, the change that helps is moving identity out of labels and into rows a query can join.
What to watch
- Whether operators on the global Meshtastic Discord who adopted the exporter publish series counts from their own networks.
- Whether a single Postgres instance keeps up as a community mesh grows past the node counts in this writeup.
- Whether anyone benchmarks the same MQTT decode workload against VictoriaMetrics or InfluxDB with metadata included.