Build1 distinct publisher3 min readPublished
The provider parses your SQL operators and emits table-level reads and writes while the DAG runs, but it emits nothing at all until you set a transport, and it files everything under a namespace called default if you skip that too.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
Dataset identity decides whether you get a graph or a pile of dots. In OpenLineage a Dataset is a namespace plus a name, a Run is one execution of a Job with its own run ID, and a Job is either the DAG or one of its tasks [7]. None of those objects holds an edge. Edges are inferred: the backend takes events fired on state transitions, START, RUNNING, COMPLETE, FAIL, ABORT and OTHER [9], and reconstructs lineage by joining datasets across runs [10]. Two tasks are linked only because they spelled a dataset the same way. The walkthrough names inconsistent dataset identity as the most common cause of a first integration that produces nodes and no edges [10], and the console transport will not catch it. Console writes the events into the task log and costs nothing to run [13], and well-formed events with disagreeing names look exactly like working ones.
You can read the shape of the resulting graph out of the demo before running it. It is one task, a `SQLExecuteQueryOperator` on a daily schedule using the `postgres_default` connection to create a table in the `analytics` schema [19]. Because the DAG is a job and each task is also a job [7], that single-task DAG registers two jobs in the backend, and a 40-task DAG registers 41 [22].
The extraction is only as wide as the parser. SQL operators need no code from you because the provider parses the query and derives inputs, outputs and column-level relationships [18]. That is the good engineering here: the lineage is derived from the one artefact that cannot misremember, the query that actually ran. It also sets the boundary. The demo is a CREATE TABLE through a SQL operator [19]; nothing in the walkthrough says what a task with no query to parse emits. So the claim that this replaces manual documentation and the lineage spreadsheet that goes stale in three weeks [21] transfers in proportion to how much of your writing goes through operators the provider can read. Three weeks is generous. If a large share of your loads are Python functions calling an SDK, facets are the extension point, since schema, SQL text, column-level lineage and your own fields all travel as facets attached to the standard objects [8].
The adoption cost is small and mostly one-time: Docker to run a backend locally [20], Python somewhere between 3.9 and 3.12 [3], one `pip install` plus a check that your image does not already ship the provider, which official Airflow Docker images may [11]. The HTTP transport wants a url and the endpoint `api/v1/lineage`, with `disabled = False` in the `[openlineage]` block [15]. After that, the number I would watch in my own context is coverage: the share of task runs emitting at least one input and one output dataset. The config lines are an afternoon. That ratio is what tells you whether anyone should trust the graph.
Ranked by verification strength, evidence, and original report placement.
SQL operators are the best place to start because the provider parses the query and derives inputs, outputs and column-level relationships without the user writing anything.
The current Airflow OpenLineage provider distribution requires Apache Airflow 2.11.0 or later, or any Airflow 3.x release, as its minimum supported version.
Two packages matter: the Airflow OpenLineage provider extracts Airflow metadata and turns it into events, and openlineage-python transmits them. The client can be upgraded independently of the provider, which is useful when you need a transport fix without touching your Airflow version.
OpenLineage has three objects and one extension mechanism: a Job is something that runs (the DAG is a job and each task is also a job), a Run is one execution of a job with a unique run ID, a Dataset is something read or written identified by a namespace and a name, and a Facet is an atomic block of metadata attached to any of the above.
Schema, SQL text, column-level lineage, run state and custom user-defined fields all arrive as facets, and the OpenLineage specification lists the standard ones.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 31, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
product
MinIO went dark on 13 February. Docker will keep patching it until 2031, for a fee.1 distinct publisher
build
The 680 MB database that was really a 17 GB disk: self-hosted support platforms fail at month six1 distinct publisher
product
AI writes the Dockerfile, and the pipeline is still checking the app code1 distinct publisher
build
Before you spend quota on an agent skill, make it pass an eval harness1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
One laptop, precise enough to check yourself
Everything rests on a single dev.to post, and no second source confirms so much as the Airflow 2.11.0 floor it asserts. What rescues the score is the kind of claim being made: pip commands, environment variables, port numbers and a complete DAG listing are all falsifiable in ten minutes by anyone who runs them, which is a stronger position than an unsourced assertion about outcomes. The one claim that is not verifiable that way — the opening promise of complete per-task lineage — is the one the piece itself undercuts later.
Nothing to count
The walkthrough never leaves localhost. One task, one Docker Compose backend, a Postgres connection named postgres_default. No team, no event volume, no number of DAGs instrumented, no report of what happens when a real warehouse's worth of runs hits an HTTP transport — so there is no adoption to measure here, only an installation procedure.
The opening oversells what the rest admits
Modestly overstated, and the overstatement is confined to the first two paragraphs. 'Every DAG run emits events naming the exact tables' is a bigger claim than 'SQL operators and many provider operators are covered, your own operators report nothing, and EmptyOperator is silent by design' — and both sentences are in the same post. Once past the promise, the piece is candid to the point of listing the failures in the order they usually appear, which is why this lands close to aligned rather than far from it.
No product behind the recommendation
Follow the money and it runs out quickly. Every component named is open source, Marquez is recommended as the reference implementation while the text explicitly allows any OpenLineage-compatible backend, and there is no vendor, sponsorship or trial link anywhere in the piece. What remains is the ordinary incentive of a developer-platform post — reputation and reach reward a clean-looking walkthrough, which is a mild pull toward the confident opening promise and away from dwelling on how much of a real DAG stays invisible.
Trustworthy today, stale by the next release
Middling, for two different reasons pulling the same way. The procedural core is easy to trust because it is easy to test, and the internal consistency of the piece — the coverage caveats, the emits_ol_events diagnostic — reads like someone who actually ran it. But version floors, port defaults and provider behaviour are exactly the facts that drift with releases, and with one source and no production sighting there is nothing here to tell you whether this survives past a local Docker stack.