Build1 publisher3 min readPublished
An iostat sample taken mid-failure showed 128 queued reads delivering 88 KB/s
Nginx reported active and ping answered while reads on one FDCServers VPS waited 18.7 seconds and I/O pressure sat near 96 percent full. The capture taken during the failure carried the ticket through a node migration to a refund.
The Engineer · Build desk
What happened
- During intermittent failures on one FDCServers VPS the guest recorded CPU iowait of 97 to 100 percent, I/O pressure near 96 percent full, reads waiting up to 18.7 seconds and flushes up to 53.6 seconds.
- Between incidents the same box measured healthy: zero processes in D state, iowait near zero, 2 to 4 ms disk latency, and both I/O pressure counters at 0.04.
- The VPS was migrated to another node, another severe guest-side stall was captured after the move, and the author then cancelled the service and was refunded.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A monitor that confirms process liveness and ping will pass cleanly through a window in which almost nothing completes, so this failure class has to be caught by a sample scheduled or triggered inside the incident.
- capability Reading /proc/pressure/io gives a tenant a measurement that does not depend on CPU accounting, so a support ticket can stand on two counters that fail together instead of one iowait number a provider can dispute.
- decision A backup overlap is a plausible one-off, so the recurrence outside that window is what converts the choice from waiting for the next scheduled job into asking for a different node.
- exposure Every number here was taken from inside the guest, and the only remedies available to a tenant holding that evidence are a migration the provider performs or an exit the provider refunds.
Localising the fault took one comparison. Called directly, the backend answered in about 3 ms, while the same requests arriving through Nginx timed out [7]. Nginx itself was blocked in filesystem reads, and EXT4's jbd2 journal thread was waiting on I/O [6]. A process whose working set is already in memory keeps answering. The process in front of it has to open files and write logs, so it drops into uninterruptible sleep, and at the worst moments 4 to 9 processes were blocked with CPU idle at 0% [8].
One iostat sample carried the rest: r_await 18744 ms, f_await 53561 ms, aqu-sz 128.54, utilisation near 100%, read throughput 88 KB/s [2]. Utilisation near 100% on its own proves nothing, because a busy device can be perfectly healthy [17]. Divide throughput by queue depth and each of the roughly 128 outstanding requests was getting about 0.68 KB/s of service [18]. In the 18.744 seconds one read waited, the whole device moved about 1.6 MB [19]. The same VPS in a healthy period showed 2 to 4 ms disk latency [3], which puts that read somewhere between 4,700 and 9,400 times slower [20].
According to the post, iowait is CPU accounting and a symptom, not a direct measurement of hardware latency [11]. "So I did not want my whole conclusion to depend on one number," the author wrote [10]. The independent signal came from /proc/pressure/io, which showed some avg10=99.14 and full avg10=95.55 during one severe period, with full later approaching 98% [9]. In the post's simplified terms, some means at least some non-idle work is stalled on I/O and full means all non-idle tasks are stalled at once [21]. Both counters read 0.04 when the machine was healthy [3].
A healthy snapshot tells you very little about an intermittent failure, the author wrote, so the system has to be captured while it is failing [16]. top, free -h, df -h and systemctl status nginx showed nothing unusual [4]. The VPS was online, Nginx reported active (running), the backend process was alive and ping worked [5].
FDCServers investigated, and one severe incident overlapped a scheduled backup [12]. That overlap is the sort of explanation that closes a ticket. The same class of storage stall then appeared outside the backup window [12], the VPS was moved to another node, and another severe guest-side stall was captured after the migration [13]. The author cancelled the VPS and FDCServers issued a refund [14].
The post does not identify what caused the stalls that fell outside the backup window. It is one guest's record of one machine. "This article is not a claim that every FDCServers server has this problem," the author wrote [15].
What to watch
- Whether FDCServers publishes a cause for the storage stalls that fell outside the scheduled backup window.
- Matching guest-side captures from other tenants on the same nodes would separate a bad node from a fleet-wide problem.
- Whether the post-migration stall ever gets a hypervisor-side explanation or stays a guest-side observation.