Build1 publisher2 min readPublished
Migrating from Dataflow to Flink includes installing cert-manager for the Kubernetes Operator
Apache Flink covers most of what Google's managed Dataflow does. A dev.to deployment guide prices the swap in prerequisites: cert-manager goes in before the operator, and the exactly-once path ends in a bucket whose keys you hold.
The Engineer · Build desk
What happened
- A deployment guide published on dev.to sets up Apache Flink on Kubernetes as an alternative to Google Cloud Dataflow, covering Session and Application clusters, exactly-once Kafka processing, Beam integration and migration.
- Its prerequisites call for a Kubernetes cluster running 1.31 or later with at least 4 CPU cores and 16 GB of RAM per node, and recommend a minimum of three worker nodes.
- The list also asks for a default Kubernetes StorageClass for persistent Kafka storage, because the exactly-once path in this design runs on a Kafka cluster you host yourself.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost The recommended node floor is capacity billed whether the pipelines are busy or idle, and the payer is whoever owns the Kubernetes account.
- decision Key rotation becomes a stream-processing concern, since whoever holds the object-storage secret key holds the recovery path for every job.
- capability A Beam pipeline can change engines without a transform rewrite. That puts the migration effort on state and delivery semantics.
- contradiction The guide asserts that self-hosting removes ongoing managed service costs, but it never prices the hardware, storage and tooling it requires.
A Flink deployment on Kubernetes starts with a controller. The Flink Kubernetes Operator extends the cluster with custom resources and automates deployment, upgrades, scaling, savepoint management and recovery [4]. It also uses admission webhooks that require TLS certificates, so cert-manager is installed before the operator [5]. The guide pins v1.21.1 from the Jetstack chart and enables the CRDs with `--set crds.enabled=true` [6].
Three worker nodes at the guide's floor is 12 CPU cores and 48 GB of RAM standing up before a single pipeline runs [8]. The workstation side adds kubectl, Helm 3, Java 11 or later and Apache Maven [16].
State is the part that has to be reproduced exactly, and Dataflow's own feature list includes exactly-once guarantees [1]. In Flink, checkpoints are automatically created state snapshots used for fault recovery and exactly-once processing [12]. The JobManager coordinates job scheduling, checkpoints, recovery and TaskManager operations, while TaskManagers execute tasks and hold state in their slots [17]. Checkpoint storage keeps checkpoint and savepoint data in durable storage for recovery and application upgrades [11], and in this guide that storage is a bucket on an S3-compatible object storage service whose name, hostname, access key and secret key you record up front [10].
Savepoints are described two ways in the same document. They are manually triggered state snapshots used for controlled upgrades, migration and operational recovery [13]. The operator, meanwhile, automates savepoint management [4]. Both can be true, since a controller can trigger and retain what a human would otherwise trigger by hand. The guide does not say which path owns a version upgrade, and a team has to settle that before the first upgrade.
The reason the swap is tractable at all is Beam. Dataflow is built on Apache Beam [1], and Flink runs Beam pipelines through the Flink Runner [3]. For the comparison to hold on a given workload, the pipeline has to be one the Flink Runner accepts, and everything Dataflow bundled has to come up as separate systems: the guide's end state lists Prometheus and Grafana monitoring, high availability, and network and RBAC security controls alongside the exactly-once Kafka path [15]. On elasticity the two lists differ by one word. Dataflow's features include autoscaling [1]; the operator's automated operations include scaling [4].
What to watch
- Whether the operator's savepoint automation covers a Flink version upgrade end to end, or a human still triggers the snapshot.
- Whether checkpoints written to a non-Google S3-compatible bucket hold exactly-once through a TaskManager failure at production throughput.
- Whether a Beam pipeline that ran on Dataflow runs unchanged on the Flink Runner, or needs transform-level rework.