A Canadian railroad says 90% of its new-table pipelines are now generated, not written

Databricks has published an account of one of Canada's largest railway networks replacing hand-authored ingestion development with code generated from a short YAML prompt, reporting more than 90% automation for new table ingestion and pipeline delivery compressed from days to minutes ^[1][4][6]. It is the first reasonably specific, numbered description of pipeline development treated as generated-by-default rather than authored, which makes it useful and also makes it worth pressure-testing.

The setup is familiar to anyone who has inherited a decades-old estate: mainframes, legacy warehouses, enterprise ETL platforms and purpose-built appliances, with hundreds of pipelines in flight during a move to a lakehouse ^[3][8]. The railroad in question spans roughly 20,000 route miles across Canada and into the United States and supports the movement of more than C$250 billion in goods a year, according to the post ^[1]. Before automation, one table meant a multi-day effort: inspect source schemas, fill in a source-to-target mapping spreadsheet, build historical and streaming ingestion, write the incremental merge, add downstream transformations, then cover schema evolution, column renames, type casts and soft deletes with tests ^[7].

The mechanics matter more than the headline number. The team combined Databricks Genie Code with custom Agent Skills, Unity Catalog for schema introspection, and a Streamlit app on Databricks Apps for source-to-target mapping ^[4]. A prompt can be as small as two lines for raw ingestion; a full table pipeline supplies source and target table names, primary keys, deduplication logic and refresh behaviour ^[12]. Generation then parses and validates the prompt, discovers historical and trusted layer schemas from Unity Catalog metadata, and auto-matches columns against source ^[13]. Output is PySpark, Spark SQL and Delta Lake, designed to run through Lakeflow Jobs ^[9].

The part worth stealing is not the model. It is that audit conventions, deduplication, change-sequence merge guards, soft-delete reconciliation and test patterns are encoded in the generation path instead of depending on each developer to remember them ^[10]. Dinesh Chandrasekaran, described in the post as Data and AI Leader at a leading Canadian transportation and logistics company, frames it as "adding determinism to a probabilistic workflow," automating what is known to be correct and leaving the interpretation layer optional ^[2][11]. That is a standards-enforcement story wearing an AI jacket, and standards enforcement is the harder half.

Now the caveats. This is a vendor blog on databricks.com about its own products, and the customer is not named ^[14][15]. The 90% figure has no stated denominator: no count of tables ingested, no defect or rework rate, no human review time per generated pipeline, no cost, no programme dates ^[16]. Ninety percent automation also means as many as one in ten new-table ingestions still needs hand work, and the residual cases in this domain are usually the expensive ones ^[17]. "Days to minutes" describes code generation, not the review, deployment and reconciliation that sit between generation and production.

What to watch: whether the railroad or Databricks publishes a table count, a rework rate, and how much of the legacy business logic survived translation intact; whether the Agent Skill approach holds when source schemas drift rather than sit still; and whether anyone reports the headcount consequence, since the stated constraint was developer bandwidth, not tooling ^[3]. If your own migration plan assumes linear staffing per table, the interesting question is not whether generation works, but what your review gate costs once it does.