Build1 distinct publisher3 min readPublished
The Tentacle scrub path walks OMAP 1024 keys at a time, and only when the scrub is deep, so a shard that crosses 200000 keys stays invisible until that PG's next deep scrub. The default interval is seven days.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
The walk is where the cost sits. Deep scrub takes a chunk of 5 to 15 objects and, for each one, steps through OMAP 1024 keys at a time, CRCing the header and then every key and every value while counting keys and summing value lengths [9][8]. Only after the walk does it compare against the two thresholds [10]. An object sitting at the default 200000-key threshold takes 196 of those steps before the comparison happens, because 200000 divided by 1024 is 195.3 [24].
Two details change how the warning text reads. "Size (bytes)" sums value lengths only, so the 1073741824-byte default is 1 GiB of values with no key bytes counted, and an object full of keys with empty values is reported at size 0 [15][27]. And `omap_checks()` stops at the first flagged object in each chunk, so a chunk holding several large objects contributes one [17]. At the default upper chunk size of 15, the count in "N large objects found in pool 'X'" can be short by a factor of 15 [26][13].
There is a boundary the source states twice and not identically. The comparison is strictly greater-than, so exactly 200000 keys is not large [16], yet the same list offers an object with 200000 keys and empty values as its example of tripping the key threshold [15]. By the rule as written, 200001 is the first large key count.
`scrub_finish()` assigns `num_large_omap_objects`, `num_omap_keys` and `num_omap_bytes` only when the scrub was deep, and nothing on the write path moves them [12][18]. Growth that lands just after a PG's deep scrub waits for the next one, which on the default interval is close to seven days of silence [28].
The published material stops short of the measurements. The stated questions include whether the OSD keeps a live per-object OMAP size anywhere queryable, which of `ceph pg dump`, `ceph df detail`, `ceph osd df`, the OSD admin socket and the Prometheus module return live numbers rather than scrub-time snapshots, what `rados listomapkeys | wc -l` costs on the OSD, and whether a supported top-N-by-keys query exists [23]. The text ends at the lab setup [29]. This evidence leaves the cost side of the early-detection techniques unresolved. The cluster is three hosts, six BlueStore OSDs, replicated size 3, running 20.2.3 Tentacle daemons under cephadm, with a single-PG test pool and autoscaling off [19][22]. Six OSDs and one PG amounts to a way to watch a deep scrub finish while you are still at the terminal, rather than a capacity test. No RGW was deployed, and `radosgw-admin` was present only to confirm command availability [20], so the bucket index question was never put to a live index.
The answer that is in the code is a design pattern. The MDS open file table caps entries per object at exactly `osd_deep_scrub_large_omap_object_key_threshold` and spreads the rest across up to 1024 objects [7], which at the default default puts its ceiling at 204,800,000 entries [25]. That is shard-at-the-application-layer, chosen by the people who wrote the check. It also means raising the OSD threshold to quiet the warning silently enlarges every MDS open file table shard, because the MDS reads the same option [7].
Ranked by verification strength, evidence, and original report placement.
When an RGW bucket index shard, a CephFS metadata object or a librados application object grows past the threshold, nothing in ceph health changes until a deep scrub happens to walk that PG.
With the default deep-scrub interval of seven days, the LARGE_OMAP_OBJECTS warning can trail the actual OMAP growth by most of a week.
The article is the result of reading the Ceph scrub code and then reproducing the condition on a Tentacle lab cluster.
ceph config get osd osd_deep_scrub_large_omap_object_key_threshold returns 200000 on the lab cluster.
ceph config get osd osd_deep_scrub_large_omap_object_value_sum_threshold returns 1073741824 on the lab cluster.
ceph config show-with-defaults osd.0 reports both thresholds with source default, so neither has been tuned on that cluster.
Distinct publishers with included, body-backed reporting in this cluster.
1 article · September 5, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
Drive-managed SMR buries the one spec that decides your rebuild window1 distinct publisher
build
The streaming edit is a software layer: proxies, stubs and an ffmpeg hook on storage you own1 distinct publisher
science
Public health's accreditor may delete the word "racism," and syllabi will follow1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
checkable in one place, confirmed in none
The specifics are unusually concrete for a single-author post: threshold values quoted straight from ceph config get, the flag condition and warning text tied to a named function, and a twenty-line librados script any reader can rerun against a test pool. Corroboration is what our coverage could not find. No upstream documentation, tracker or second operator appears alongside dev.to, and the text breaks off in the middle of the ceph df detail output that was meant to show the live pool-level counter.
just a rig, no fleet in sight
Three hosts, six OSDs and no gateway describe a reproduction environment. The bucket index shard that opens dev.to's write-up is the one case never deployed, and there is no operator disclosure, reader report or production incident anywhere in this reporting to indicate how often this bites real clusters or at what size.
buried under its own bullets
The headline promise stays inside what the code trace shows, and no claim reaches past the lab. If anything the framing undersells: the two findings an operator would most want up front, a 200,000-key object reported at Size 0 and a count that can run fifteenfold low, sit as bullets under 'details worth knowing' after the diagram.
no product, no pitch
It reads as a personal write-up with no product at the end of it. We didn't find any vendor tooling recommendation, any configuration change favouring a particular setup, or a paid monitoring pitch. The one pull worth naming is the genre, since a deep-dive post rewards a clean narrative, and the question list at the top promises answers dev.to's published text does not deliver.
verifiable, but only by the reader
The mechanism claims can be falsified in an afternoon by anyone with a test pool, which keeps this well off the floor: function names, the strictly-greater comparison, the 1024-key step. It stops mid-range because one author's code reading is the whole chain of custody, the behaviour is pinned to 20.2.3 with nothing checked against other releases, and the live-query results the author set out to gather are not in the text we have.