Zero-Instrumentation Go Tracing Works, But You Are Now Maintaining ABI Offsets

A writeup on dev.to by Neeraj Singh argues that attaching eBPF uprobes to Go runtime symbols and to net/http and gRPC entry points is now an operationally viable way to reconstruct distributed traces with no code change in the target binary ^[2]. That matters because the cost it removes is real -- every SDK call site, context propagation branch and baggage extraction is code that can drift, be omitted in a hot path, or add measurable CPU at high RPS ^[1] -- but the same piece names three Go-specific failure modes that replace it ^[18].

The mechanism is old and blunt. A uprobe patches a breakpoint instruction at an offset in a running binary; when execution reaches it the kernel pauses the thread, runs the BPF program, and resumes ^[3]. For C or Rust that maps cleanly onto a function prologue ^[4]. Go does not cooperate.

First, correlation. The M:N scheduler multiplexes goroutines onto OS threads, so a request handled by goroutine G on thread M1 when the entry probe fires may be rescheduled onto M2 before the response is written ^[5]. A probe that keys on pthread_self() or the current PID/TID loses continuity across that yield ^[6]. According to the author, the correct anchor is the goroutine ID in the runtime.g struct, which means either a BTF-aware map keyed by goid pulled off the goroutine stack, or a fixed-offset computation against the g pointer held in the thread-local storage register: FS on amd64, R28 on arm64 ^[7]. That is not a one-time integration. It is a structural dependency on runtime internals.

Second, the ABI. Go 1.17 moved function arguments into registers -- AX, BX, CX, DI, SI, R8 through R11 on amd64 -- instead of the stack ^[8]. A probe written against the 1.16 convention that reads ctx from sp+8 returns garbage on 1.17 and later ^[9]. The stated options are shipping ABI-aware probe logic per Go minor version or resolving DWARF location expressions from the binary's debug info at each probe site ^[10]. Choose the second one, or you have quietly signed up for a compatibility matrix.

Third, the compiler. Go inlines small functions aggressively, so if http.(*Transport).roundTrip is partially inlined the symbol may not exist where the tracer expects ^[11], and probing the wrong offset yields missed spans or corrupted argument reads with no error ^[12]. Silent is the operative word.

The plumbing is the least interesting part: probes on HTTP and gRPC entry and exit, on runtime.newproc1 and runtime.goexit for goroutine lifecycle, and on crypto/tls handshake functions for latency attribution ^[13], feeding a BPF ring buffer -- preferred over perf buffers on kernels 5.8 and above for overhead and ordering ^[14] -- into a cilium/ebpf consumer whose event struct must match the C definition's alignment padding byte for byte, or every field after the first decodes wrong ^[15].

Then the hard problem: propagating context across service boundaries when you cannot inject a header in application code ^[16]. The option described attaches a tc hook and writes W3C traceparent bytes into the packet with bpf_skb_store_bytes, which needs CAP_NET_ADMIN and works only for cleartext HTTP/1.1, because TLS terminates above the socket layer and the program sees ciphertext ^[17]. So the propagation path excludes exactly the TLS-heavy services the crypto/tls probes were added to measure ^[19].

Watch three things before this goes into a production stack: whether your tracer resolves offsets via DWARF or hardcodes them per Go version ^[10], whether your kernel floor is 5.8 ^[14], and whether span counts on a service you have also instrumented by hand agree, since inlining misses do not raise errors ^[12]. The source material available here breaks off at the start of the second propagation option, so the non-packet path is unresolved.