Build 1 publisher2 min readPublished September 21, 2026 at 8:14 PM PDT

An NFS-only StorageClass leaves Zeebe's RocksDB commits without an atomic rename

An air-gapped cluster of 10 servers runs every workload on NFS. The write-up blames that for Elasticsearch and Zeebe I/O failures, and its own numbers rule out Longhorn and local-path on ~100GB node disks.

The Engineer · Build desk

Illustration accompanying An NFS-only StorageClass leaves Zeebe's RocksDB commits without an atomic rename

What happened

A self-managed Kubernetes cluster in an air-gapped private datacenter runs 10 servers with roughly 100GB SSDs each, and NFS is the only StorageClass available to any workload on it.
The write-up says that setup is adequate for general workloads and fails for the I/O-intensive ones, naming Elasticsearch and Camunda's Zeebe as the applications it cannot serve.
Elasticsearch data has passed 100GB and keeps growing, which the post gives as the reason the cluster's limited local disk rules out Longhorn and local-path.
The air gap removes AWS EBS and Azure Disk from consideration, and the post says the DevOps team is left with no straightforward alternative to NFS.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

constraint A single NFS StorageClass sets the admission list for the whole cluster: any product whose storage engine needs atomic rename and a real fsync has to wait for a second class to exist before it can be scheduled at all.
decision Every path off NFS here starts with hardware someone has to buy and rack inside the air gap, so the next move is a procurement decision taken by whoever owns the datacenter.
exposure The risk the post describes is carried by the teams that own Zeebe and Elasticsearch, who would see it as corrupted application state and lost writes while the storage layer reports success.
contradiction The post's case against NFS is about POSIX semantics, but its case against the local-storage options is about disk size, and those two constraints have different remedies.

Zeebe keeps its state in RocksDB ^[3]. RocksDB commits by renaming a file, and the dev.to write-up argues that NFS's stateless protocol does not guarantee atomicity across a network round trip; it attributes partial writes and data corruption to that gap ^[5]. The Elasticsearch half of the argument turns on fsync. According to the post, NFS satisfies sync inconsistently and often from the server's cache instead of the underlying storage, so a write can be acknowledged before it is durable, and the post ties write amplification and delayed persistence to that behaviour ^[6]. It also counts round trips: a stateless protocol needs one per metadata operation, which the post says accumulates into latency and head-of-line blocking on indexing and state-management paths ^[7].

Those are statements about what the engines require ^[4]. The post does not report latency measurements, error output, or the NFS version and mount options in use ^[15]. For the failure to transfer to another cluster, that export would have to be behaving the way the post assumes: sync answered from the server's cache, and rename not atomic under concurrent access. Those are properties of one server and one mount, and a test export against that server would settle both questions.

The capacity half is arithmetic on the post's own numbers. Ten servers at about 100GB of SSD each is roughly 1TB of raw local disk across the whole cluster ^[10]. The Elasticsearch data is already past 100GB and still growing ^[8], so one copy of it claims about a tenth of every raw byte the cluster owns ^[12].

Adding servers does not help the way it looks like it should. Ten more identical machines take aggregate raw capacity to about 2TB ^[13], and each individual disk is still 100GB, below the size of a single copy of the index ^[11]. The post excludes Longhorn and local-path on that disk size ^[8]; it excludes AWS EBS and Azure Disk on the air gap ^[9].

The consequence the post states for staying put is sustained performance degradation, data integrity risk and a higher probability of downtime ^[14].

What to watch

Whether the team publishes the NFS version, server implementation and mount options, which would show whether these failures are protocol-level or configuration-level.
Whether the promised solutions section names an on-premises block target and the per-node capacity it assumes.
Whether Elasticsearch growth forces a disk purchase before any StorageClass change lands.

Claim ledger

Ranked by verification strength, evidence, and original report placement.

[1]
The environment is a self-managed Kubernetes cluster in an air-gapped private datacenter, consisting of 10 servers with roughly 100GB SSDs each, using NFS exclusively as the primary StorageClass.
ReportedSupportedView cited source
[3]
RocksDB underpins Zeebe's state management.
ReportedSupportedView cited source
[8]
With Elasticsearch data exceeding 100GB and continuously growing, the cluster's limited local disk capacity precludes the use of local storage solutions like Longhorn or local-path.
ReportedSupportedView cited source
[9]
The air-gapped nature of the environment eliminates cloud-based block storage options such as AWS EBS or Azure Disk, leaving the DevOps team with no straightforward alternatives.
ReportedSupportedView cited source
[15]
The post presents no latency measurements, no error output, and does not name the NFS version, server implementation or mount options in use.
ReportedSupportedView cited source
[10]
Aggregate raw local SSD capacity across the cluster is about 1TB.
DerivedSupported1 source— sign in to open
[11]
One copy of the Elasticsearch dataset is larger than any single node's entire disk.
DerivedSupported1 source— sign in to open
[12]
One copy of the Elasticsearch dataset takes about 10 percent of the cluster's aggregate raw local capacity.
DerivedSupported1 source— sign in to open
[13]
Ten more servers of the same specification would take aggregate raw local capacity to about 2TB while leaving each individual disk at about 100GB.
DerivedSupported1 source— sign in to open
[2]
The NFS-only configuration adequately supports general workloads but fails to meet the demands of I/O-intensive applications such as Elasticsearch and Camunda (Zeebe).
ReportedInsufficient2 sources— sign in to open View cited source
[4]
The post says NFS lacks support for the atomic operations RocksDB requires to guarantee data consistency, specifically rename, fsync and file locking.
ReportedInsufficient2 sources— sign in to open View cited source
[7]
The post says NFS, as a stateless protocol, requires a network round-trip for each metadata operation such as file open, seek and write, introducing cumulative latency and head-of-line blocking that degrades throughput for Elasticsearch indexing and Zeebe state management.
ReportedInsufficient2 sources— sign in to open View cited source
[14]
The post says the consequences of inaction are sustained performance degradation, data integrity risks and increased downtime probability without a block storage solution.
ReportedInsufficient2 sources— sign in to open View cited source
[5]
The post says RocksDB employs atomic rename operations to commit transactions, that NFS's stateless protocol introduces race conditions during rename because atomicity is not guaranteed across network round-trips, and that this results in partial writes or data corruption.
ReportedInsufficientView cited source
[6]
The post says Elasticsearch relies on fsync to ensure data is persisted before acknowledging writes, that NFS's implementation of sync is inconsistent and often delegates the operation to the server's cache rather than the underlying storage, and that this causes write amplification and delayed persistence while increasing the risk of data loss during failures.
ReportedInsufficientView cited source

Sources

Publishers with included, body-backed reporting in this cluster.

dev.to
1 article · September 21, 2026
NFS Storage Causes I/O Issues in Air-Gapped Kubernetes Cluster: Block Storage with POSIX Semantics Needed

Ask Clarity

Have a question this piece left open? The answer comes from its reporting and claim ledger — nothing else.

Try

Clarity analysis

How the coverage leans and what the record actually supports. Read the methodology

Perspective Coverage

How much of the coverage speaks to builders, operators and investors. It measures emphasis, not whether the story is right.

The claim chain is the useful part: RocksDB commits by renaming a file, Elasticsearch acknowledges a write once fsync returns, and NFS is said to weaken both. Whether that holds on these ten servers depends on the NFS version and the mount options, but dev.to's post does not say which version or which options were in use. Locking and durability behave differently between NFSv3 and NFSv4, so a builder reading this cannot tell whether the fault described would appear on their own mount.

Reality

Evidence, adoption, hype gap, incentives and confidence, each read independently from the current record on the scale shown beneath it. How these are measured.

Evidence28

0100

First-hand setup, asserted diagnosis

The hardware picture is first-hand and easy to believe: ten servers, ~100GB SSDs, one StorageClass, no route out, but the diagnosis built on that picture remains an assertion rather than a finding. There are no latencies and no error output, and the post does not say which NFS version or mount options were in use, so a reader cannot tell whether the corruption risk described belongs to NFSv3, to a mount setting, or to something else. The capacity conclusions are the sturdiest part of the piece, because they follow from the author's own figures.

Adoption12

0100

One site on record; the remedy is still on paper

The only deployment on the record is the author's: NFS underneath Elasticsearch and Zeebe, in one air-gapped datacenter. The block storage he argues for has not been installed, benchmarked or costed, so the claimed improvement has nothing behind it yet.

Hype gap+42

Topics and entities

Follow any of these and your For You feed starts watching them — no settings page required.

Topics

Entities

build
The exactly-once guard that survived 40 racing processes broke on its own cleanup path
1 publisher
build
A label selector now freezes the Postgres data and WAL volumes at the same instant
1 publisher
build
SSE in Go breaks twice before your handler runs: an illegal header, then a 30-second timeout
1 publisher
build
Rate limit your MCP servers, because a retrying agent turns one error into a billing incident
1 publisher

What happened

Why it matters

What to watch

Claim ledger

Sources

Let Clarity write the post for you.

Topics and entities

Topics

Entities

Related stories

Claim ledger

Sources

Let Clarity write the post for you.

Topics and entities

Topics

Entities

Related stories

Perspective Coverage

Reality