Your slow box already reports runqueue wait per thread. Do not turn it into a percentage

A writeup published on dev.to takes one 16-core Linux host through a quiet regime and a saturated one, sampling per-thread runqueue wait out of /proc/<pid>/task/<tid>/schedstat, a file that is already on every host you run ^[1][7]. The useful result is not the file; it is that the obvious way to normalise it inverts the ranking, putting an idle kernel thread ahead of application threads that were actually denied a core ^[11][16][17].

The file holds three numbers: nanoseconds this thread spent on the CPU, nanoseconds it spent waiting on a runqueue while runnable, and the count of timeslices it ran ^[1][2]. On PID 1 of the test host that read as 297508187061 22806177929 768442 ^[3]. Field two is the operationally distinct one, because it is not blocked on I/O, not sleeping, not idle: it is ready to run and denied a core, which the author calls the only kind of waiting a bigger machine would have fixed ^[4]. That is the line between the two failure modes operators keep conflating. A thread parked on a socket read contributes nothing to field two, and adding cores will not move it. A thread with seconds in field two is telling you there was more runnable work than cores.

Nothing standard reports this per thread. /proc/pressure/cpu says how much of the last ten seconds some task spent stalled on a runqueue, and top says who is burning CPU; neither names the thread that wanted the CPU and did not get it, and as the author puts it, "The one burning CPU is winning" ^[5][6]. The instrument is two snapshots of every thread's schedstat, a sleep, and a subtraction, printed as wait seconds, run seconds, and a fraction, sorted both ways ^[8].

The fraction is wait/(wait+run) ^[9]. On the quiet machine, load average 5.12 across 16 cores, roughly 32 percent of core count, the top thread by fraction was ksoftirqd/10 at 94.7 percent ^[10][23][11]. Its wait was 0.003 seconds over a three-second window, with run time too small to print at millisecond resolution ^[11][12]. Machine-wide, all 85 waiting threads together booked 0.018 seconds of runqueue wait ^[13].

Then four concurrent whisper.cpp transcriptions, eight threads each: 32 runnable threads on 16 cores, two times oversubscribed, load average 11.14 and climbing ^[14][22]. All four runs returned byte-identical correct transcripts of the JFK sample, so this was real work rather than a spinlock imitating one ^[15]. The top three threads by absolute wait sat at 2.063, 2.013 and 1.972 seconds of queue time, against 1.075, 1.147 and 1.190 seconds of run time ^[16]. Those three alone accumulated 6.048 seconds of runqueue wait in a three-second window, about 336 times the entire quiet machine's total ^[19][20]. The worst of them spent 69 percent of wall clock runnable and not running ^[21].

Their fractions were 65.7, 63.7 and 62.4 percent ^[16] - below the idle ksoftirqd. The author's summary: the starved thread waited 690 times longer and scored 29 points lower, so an alarm keyed on the fraction fires continuously on an idle box and quietens when work actually starves ^[17][18]. The denominator is the problem. A thread that wakes, queues briefly, runs for microseconds and sleeps pins the ratio near 100 percent; threads doing sustained work dilute it with run time.

What to watch: page on absolute wait-seconds per window, divided by the window rather than by run time, and aggregate it by process or cgroup before it reaches a dashboard. Then check whether your own hosts expose the field at all, and whether the reading is stable across sampling windows, because all of the above comes from one 16-core machine in one writeup ^[7].