Skip to content

Build1 publisher3 min readPublished

Nine BPF instructions move HTTP instrumentation from the deploy pipeline to the host

A single-file Python agent taps the interface with the same filter tcpdump prints for tcp, and it reconstructs method, path, status and p95 latency for HTTP/1.x while gRPC inside TLS stays an opaque byte count.

The Engineer · Build desk

Illustration accompanying Nine BPF instructions move HTTP instrumentation from the deploy pipeline to the host

What happened

  • The agent opens a raw AF_PACKET socket, which on Linux receives a copy of every frame crossing any interface, the same tap tcpdump uses, and then attaches a classic BPF filter with setsockopt(SO_ATTACH_FILTER).
  • The filter is the nine-instruction program tcpdump -dd tcp prints: it accepts IPv4 and IPv6 frames whose next header is TCP and drops UDP, ARP and ICMP in the kernel, before a byte reaches user space.
  • There is no stream reassembly, because an HTTP/1.x request line starts a client segment, a status line starts a server segment, and a TLS ClientHello is the first record on a fresh connection.
  • Paths are normalised so numeric segments become {id} and UUIDs {uuid}, then aggregated per interval into count, 4xx, 5xx and p50/p95/max per direction, host, method and path.
  • The whole thing is a single Python file using only the standard library, so it runs on any Linux with Python 3.8 or newer.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • decision Turning on per-request capture is now a question for whoever holds root on the box, and a team that cannot get an SDK merged into fourteen services can still get endpoint latency for all of them.
  • constraint Anyone whose internal calls are gRPC gets flows and byte counts from the tap, so they keep paying for the proxy access log path or the in-process SDK anyway.
  • capability Because SNI arrives in the clear, an ops team can produce a list of the third-party endpoints a server actually calls without reading the application code.
  • exposure Requests whose Host header lands past the first 2 KB drop out of the parse, so the per-endpoint error rate describes parsed traffic and not all traffic.

What you get depends on what your services speak. Plain HTTP/1.x yields method, path with the query string stripped, `Host`, status code, bytes each way, and the time from the request segment to the first byte of the response [6]. The post says that is enough for per-endpoint counts, 4xx and 5xx rates and p50/p95/max [6]. HTTP/2 multiplexes streams inside one connection and is almost always inside TLS, so without keys it shows up as an opaque flow with byte counts; gRPC is the same, and cleartext `h2c` is binary-framed, so the request-line parser skips it too [12]. Group the traffic cases the post enumerates and two give request-level detail, one gives host-level metadata from SNI, and three give byte counts or nothing [23].

The post says traffic between a reverse proxy and the app on `127.0.0.1:8000` is usually plain HTTP, so the tap gets full request-level detail for your own application even when the public side is TLS [11]. Outbound calls have to be TLS 1.2 or 1.3, where the ClientHello carries Server Name Indication in the clear [8]. Encrypted ClientHello hides SNI, though the post says adoption on API endpoints is still rare [10]. Among what the outgoing map turns up, the post lists "a third-party API nobody knew about" [9].

Two parser decisions are better than they had to be. Keep-alive connections that pipeline requests are attributed by order, and the post notes that is correct for HTTP/1.1 because responses must arrive in request order [15]. Absolute-form request targets, the kind proxies send, are normalised to a path [7]. The limit sits in the first segment: a request is only parsed if the request line and the `Host` header are within its first 2 KB [14]. Large headers on a long URL can push `Host` past that window, and the post does not say whether the agent counts what it failed to parse.

The SDK path costs a redeploy of every service and a bet that your framework version is supported [22]. The tap costs root. Installed as root it enables capture automatically; without root it still reports metrics, processes and connections from `/proc/net/tcp`, but no per-request detail [21]. That will be either the shortest install of your year or the longest conversation with your security reviewer. This is the right trade for a fleet of HTTP/1.1 services behind nginx; for a gRPC mesh it is not, because the tap counts bytes and the request detail has to come from the proxy access log or an in-process SDK [12][13].

The agent keeps a small sample of the slowest and failing requests with their timing [18]. Each destination's reverse-DNS name is resolved once and cached, so the outgoing map shows `api.stripe.com` instead of an IP [19].

What to watch

  • Whether the agent double counts a request it sees on loopback and again in the nginx access log it tails from the standard path.
  • Framework defaults moving internal calls to HTTP/2 or gRPC, which would shrink the share of traffic the request-line parser can read.
  • Any move to per-request HTTP/2 detail, which needs TLS keys or an in-process hook, not a better filter.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories