Build1 publisher3 min readPublished
One GCP VM runs an ML team's Airflow for under $150 a month against Composer's $350 estimate
One ML team moved Airflow off on-prem boxes onto a single GCP VM estimated at under $150 a month, against about $350 for Cloud Composer 3. Its published repo gives other teams a working build to price against their own Composer quote.
The Engineer · Build desk
What happened
- The team's organization is phasing out its on-prem setup and going fully cloud native, with the team's own pipelines and compute moving into GCP.
- The team built a Cloud Composer 3 environment with Terraform first, pointed its DAGs at it, and it worked.
- The replacement is one Compute Engine VM running self-managed Airflow 3.x with Postgres, provisioned per environment with Terraform and maintained with a Fabric script.
- After running the Composer and VM builds side by side, the team chose the self-managed VM.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost The saving of at least $200 a month has to pay for the engineer hours now spent upgrading Airflow and patching the VM, work Composer would have covered.
- constraint The price only carries over to teams whose Airflow hands compute to other services; DAGs that run heavy tasks on the worker would outgrow one small VM.
- exposure Losing the one VM or its disk would take down the scheduler and, apparently, its metadata database together, so the restore procedure now matters more than it did across several boxes.
According to the team's dev.to write-up, Composer 3 bills for compute units every hour the environment is alive, whether a DAG runs or not [6]. A scheduler spends most of its day waiting for the clock, so on that meter it mostly pays for idle time. This team's Airflow decides when a job runs and hands the job to Vertex AI or BigQuery. The author concluded that a scheduler like that fits on one small VM [7].
Both prices are estimates: the $350 came from Google's pricing calculator for a small Composer environment [5], and the under-$150 figure is the team's own estimate for the VM build [9]. The gap is at least $200 a month, or $2,400 a year [1][2]. That number transfers to a team whose DAGs mostly submit jobs to other services. If tasks do their compute inside Airflow workers, one small VM is the wrong size, and I'd expect the gap to shrink as the machine grows.
About $20 of the estimate is an HTTPS load balancer [10]. GCP charges a flat $0.025 an hour for the first five forwarding rules, and the author puts that at roughly $18 a month plus data [10]. Over a 730-hour month it comes to $18.25 [4]. The balancer fronts config_hq, a Flask app on Cloud Run where users save pipeline settings to a GCS bucket, and it carries the IAP login [15]. The VM, its disk and Cloud NAT share the remainder, under about $130 [11][3].
The network design is the part I would copy. The Airflow VM has no external IP [16]. Operators reach SSH through an IAP tunnel on port 22 with OS Login, and the Airflow UI through an IAP tunnel on port 8080 [16]. Outbound traffic goes through Cloud Router and NAT to github.com, pypi.org and dev.azure.com [17]. The fetch_configs DAG reads the config bucket with objectViewer. A wheel-build script uploads to the artifacts bucket with objectAdmin, and the VM submits Vertex AI training jobs as its own service account [18].
Reliability is harder to check from the post. The stated goal was to end up no less safe or reliable than before [3]. Before, Airflow ran on a few hand-installed on-prem Linux boxes [1]. Now it runs on one VM [8]. The cost breakdown has no managed database line, so the Postgres metadata store appears to share that VM [5]. The published excerpt does not describe backups or failover for it.
Patching moves too. With Composer, Google upgrades Airflow and patches the machine; on the VM, the team does [12]. The author wrote that "we are a technical team that already ran Airflow ourselves, so the extra work was work we knew" [14]. The code is in the self-managed-airflow-on-gcp repo, and the post walks through a trimmed-down working version of it [19].
What to watch
- The team's actual GCP invoice for the VM build after a full month, set against the under-$150 estimate.
- Whether the self-managed-airflow-on-gcp repo adds Postgres backups or a documented restore procedure for the single VM.
- The first Airflow 3.x upgrade the team performs on the VM, the first real test of the patching work Composer would have handled.