Build1 publisher2 min readPublished
The opentelemetry-instrument launcher patches dagster.asset before definitions.py imports it
opentelemetry-instrumentation-dagster hooks Dagster's decorator factories from the standard OTel launcher, so every op and asset emits a span with no imports in the pipeline file. Kubernetes step Pods need the hook baked into the image.
The Engineer · Build desk
What happened
- The existing dagster-otel package traces pipelines through a @traced() decorator stacked under @op or @asset, which is opt-in and requires editing every function you want traced.
- Its author has built a companion package, opentelemetry-instrumentation-dagster, that emits spans for ops and assets with no decorators and no imports in the pipeline files at all.
- Turning it on is a pip install plus running the existing Dagster command through the opentelemetry-instrument launcher with OTEL_SERVICE_NAME and OTEL_EXPORTER_OTLP_ENDPOINT set.
- Each patched decorator factory wraps the incoming compute function with dagster_otel.traced() before passing it to the real decorator, so Dagster builds its definitions from already-traced functions.
- For k8s_job_executor the author's fix is static: copy sitecustomize.py into the site-packages root when the container image is built, so every step Pod picks it up at interpreter startup.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision Choosing between the two packages is now a choice about selectivity: the decorator lets you pick which functions emit spans, and the launcher gives you all of them with no per-function say.
- constraint On Kubernetes, tracing stops being a runtime environment variable and becomes a property of the image, so a team that cannot rebuild and redeploy the image cannot switch it on for a debugging session.
- capability Linking a run to its trigger, whether a sensor tick, a CI pipeline or an upstream OTel-instrumented service, becomes possible for pipelines whose owners will not accept a tracing import in their code.
- exposure Span coverage now rides on four Dagster call signatures that the pipeline code itself never references, so an upstream change there breaks tracing everywhere with nothing in definitions.py to point at.
Import order decides whether any of this works. The manual decorator wraps a function you wrote, so timing never matters. The automatic version has to replace `dagster.asset` before your `definitions.py` executes `from dagster import asset`, because after that line the module holds its own reference and patching the name in the `dagster` namespace reaches nothing [8]. The launcher handles that generically. It discovers every package registered under the `opentelemetry_instrumentor` entry point and calls `.instrument()` on each before your code imports anything [9].
There was an easier route, and the post explains why it was dropped. Reaching into an already-built `AssetsDefinition` and swapping its compute function means mutating a non-public attribute of an object that was never meant to change after construction [7]. The shipped package patches the decorator factories instead, which the post describes as public, stable API [5]. Because the wrap happens on the way in, Dagster builds the `OpDefinition` or `AssetsDefinition` from an already-traced function [6]. I would take that trade: a public call signature changes with a deprecation cycle, a private attribute changes whenever.
Cross-process execution is the harder case. Dagster's `multiprocess` executor starts a fresh interpreter per step and re-imports the Definitions module from scratch, so a patch applied only in the parent is not there [10]. `opentelemetry-instrument` already covers that: it inserts a directory holding a two-line `sitecustomize.py` at the front of `PYTHONPATH`, then `execl()`s into the real command [11]. Python imports any module named `sitecustomize` it finds on `sys.path` at interpreter startup, and spawned subprocesses inherit `PYTHONPATH` by default, so each child re-triggers instrumentation on its own [12].
Kubernetes breaks the inheritance. Under `k8s_job_executor` each step is a genuinely separate Pod, and `PYTHONPATH` is not among the env vars Dagster forwards into it [13]. The post says the OpenTelemetry Operator's own Kubernetes auto-instrumentation has the same shape as the static fix [15].
The post is plain about what the zero-code version costs. "Auto-instrumentation is the opposite trade-off: zero code changes, in exchange for monkeypatching and losing per-function visibility," its author wrote [16], and points at the OpenTelemetry Python ecosystem's own split, where `opentelemetry-instrumentation-flask` and `-django` ship as separate packages from the manual API and SDK instead of as toggles on it [17]. One coverage question sits inside the post's own two lists. `@dbt_assets` appears among the decorators that get a span, while the four patched factories named are `dagster.op`, `dagster.asset`, `dagster.multi_asset` and `dagster.asset_check` [18]. The post does not say how the dbt case is covered.
What to watch
- Whether a later release names dbt_assets among the patched factories, closing the gap between the post's two lists.
- Whether Dagster starts forwarding PYTHONPATH into step Pods, which would remove the need for a baked-in sitecustomize.py.
- Whether the signatures of dagster.op and dagster.asset change in a Dagster release, since the patch calls through them.