Forty-eight hours of load balancer forensics ended at a module-level requests.Session

The two things being compared are maintained by different processes. A balancer reports on the targets it registered and the connections it is prepared to open. The socket that kept returning the deleted fields was a file descriptor inside a Python process that had been running since before the deploy ^[10], and the target view does not enumerate those ^[2].

Connection reuse is what hides the old code path. Once a TCP connection is idle but still alive, the next request goes back down the same connection and the resolver is never asked again ^[9]. For as long as the pool holds that entry, it makes no difference that the hostname now points at new instances ^[5].

Every check that came before the client interrogated either the balancer or a brand new connection ^[1]. The laptop looked innocent for the same reason: curling the public hostname returned the new header every time ^[6], because a short curl opens one connection, reads the body and exits ^[7]. Each of those invocations exercised the one path that could not fail, a connection opened after the cutover ^[3]. "I blamed eventual consistency out loud, which is usually how I mark a session that has already gone sideways," the author wrote on dev.to ^[11].

A working repro needs a server that prints a generation string, and a way to replace that generation without changing the hostname ^[15]. In the post's whoami_server.py that is an http.server subclass with protocol_version = "HTTP/1.1", commented "keep-alive on", plus an explicit Connection: keep-alive header on every response ^[14]. The client side is one line at module scope, session = requests.Session(), then a thread that loops a GET with timeout=(0.5, 2.0) and sleeps a second between calls ^[12]. The one-second gap is short enough that the pool entry stays in use for the whole life of the process ^[6].

For this to be your bug, three things have to hold. The client is a long-lived process holding a session created at import time and passed around like a global database handle ^[8]. The connection was opened before the cutover ^[10]. And the host has to keep idle sockets alive across it: the post says to run the local server where the workers actually run, and not to trust a laptop TCP stack to keep idle sockets alive as long as Linux does ^[16]. The evidence is that repro and the author's account of the debugging loop; the post calls itself field notes, and it comes without production dashboards or graphs ^[17].