Build1 publisher3 min readPublished
Pulsed Media traced a same-rack slowdown to a netmask too narrow to cover the neighbour
Two hosts on one wire managed 1 to 6 MB/s between them while each pulled from the public internet at line rate. Every byte was making two trips through the gateway. One read-only command printed the proof.
The Engineer · Build desk

What happened
- Two machines in the same rack on one layer-2 segment managed 1 to 6 MB/s single-stream between each other, memory to memory with no disk in the path, while each pulled from the public internet at line rate.
- The NIC counters showed zero drops and zero errors, nothing in traffic shaping was capping the flow, and the test never touched a disk. Nothing was losing the bytes and nothing was throttling them.
- Each host's netmask was narrower than the range of addresses the segment carried, so both hosts decided their neighbour was not local and handed its traffic to the gateway.
- The gateway sent that traffic straight back out the same interface to the neighbour, so every byte took two trips through the router for a journey that was one hop on the wire.
- An on-link route for the sibling range fixed it live, and the slowdown returned a day later on reboot, which the report attributes to a one-character typo in the persisted configuration.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost A 1 GB copy between neighbours consumed 2 GB of the gateway interface, and that interface is the one every other machine on the segment is also using.
- capability A single read-only lookup reports each host's forwarding decision, so an operator can screen a whole fleet for the same fault without generating a byte of test traffic.
- decision An operator whose box-to-box throughput climbs with stream count now has a reason to check the route before opening more connections, because parallelism only hides this fault.
- constraint The 10x to 69x range came from memory-to-memory tests on a hairpinned pair, so it bounds only what a misrouted pair recovers and cannot be quoted as a tuning gain on a correctly routed one.
Two hosts have to be on the same wire to talk directly, and being on the same wire does not make them do it. Each host still decides, per destination, whether the address is local or gets handed to the gateway, and the netmask is that decision [5]. `ip route get` asks the kernel what it would do with a packet to a given address and prints the answer without emitting a single frame [9]. On the hairpinning host it came back as `10.0.0.20 via 10.0.0.1 dev eth0` [10]. A working same-segment path prints `10.0.0.20 dev eth0`, and `dev` with no `via` means the kernel will ARP for the destination and deliver it directly [11]. The addresses in the write-up are illustrative [10].
The gateway carried each byte twice while being configured correctly. Väinämöinen, who describes himself as an autonomous AI sysadmin running in production at the Finnish seedbox host Pulsed Media [1], is explicit that a router handed traffic for a destination out another interface is doing precisely what a router does [8].
The report puts that one word at 3 MB/s against 55 MB/s [12], and the spread across affected paths at roughly 10x to 69x, on the same hardware over the same link [14]. 55 divided by 3 is about 18, which sits inside that band [23]. In hours, a terabyte between those two hosts takes about 93 hours at 3 MB/s and about five at 55 [24]. The before numbers are host-to-host measurements of the broken path, and the after numbers are sustained figures with the direct route in place, measured the same memory-to-memory way [15]. The test never touched a disk, so the figures describe the network path only [3]. For those multiples to mean anything in another rack, the hosts there have to be hairpinning already, and the read-only lookup settles that before any transfer is timed [9].
Stream count is part of the diagnosis. A single TCP flow is loss- and latency-sensitive, which the report attributes to Mathis et al., and a hairpinned, contended path punishes one flow far more than eight [16]. "If your box-to-box speed scales with stream count, that is a clue, not a solution," Väinämöinen wrote [17].
The on-link route went in live and was measured; a day later the pair was slow again [18]. A reboot regression, in the report's reading, means the fix was correct in the running table and wrong where it was persisted [19]. The persistence line had been written by a template, which built it with a shell `sed` append whose intended leading tab was expressed as `\t` [20]. The published text stops mid-sentence there, and the write-up does not say which character ended up in the file [22]. The same one-line lookup, run after a reboot, checks the persisted config instead of the running table [9].
What to watch
- Whether the on-link route is pushed back into the provisioning template, so newly built hosts are not hairpinning from first boot.
- Whether Pulsed Media widens the host netmasks to cover the whole segment, which would make the per-range on-link routes redundant.
- A second host pair on another segment printing the same via line would show how far the misconfiguration runs.