Build1 publisher3 min readPublished
Hybrid DR puts an always-warm inference stub in a cluster scaled to near-zero
The web tier's recovery target is 15 minutes; the on-prem ML pipeline's is 24 hours. A dev.to writeup covers the gap with a smaller model kept deployed in a second Azure region, and that stub runs continuously.
The Engineer · Build desk
What happened
- A hybrid design keeps web application layers on AKS while 50-plus ML inference services stay on-premises, held there by data residency rules, sunk GPU capex and models trained on data that cannot leave the network.
- Failure now spans a cloud region, a data center that can fail independently of it, and the network link between them, each able to go down on its own.
- The four-tier ladder is adapted to this topology, but the available text details only Backup and Restore and Pilot Light, with Warm Standby and Multi-Site listed by name and cost label.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Pilot Light is priced on the assumption that only skeleton infrastructure runs continuously, and a hybrid estate breaks that assumption by keeping an inference stub warm in the standby region every hour of the month.
- constraint Graceful degradation has to live in the calling code, so every AKS service that consumes an inference endpoint needs a branch for the stub's answers. That is application design work, not a runbook entry.
- decision With no on-prem failover site in the design, the choice for the ML estate stops being which DR tier to fund and becomes how much model quality the product will accept during an outage.
- exposure Anyone copying the pattern commits 50-plus services to a degraded mode of unstated quality, because the accuracy loss of the quantized stub is not quantified in the post.
The Pilot Light tier here is a secondary AKS cluster in a second Azure region, defined entirely as code in Bicep, Terraform or ARM templates, scaled to near-zero: one system node pool, no user workloads running [15]. Namespaces, secrets through Azure Key Vault and the CSI driver, the VNet and private endpoints are all pre-provisioned [16]. The cost case for the tier is that only skeleton infrastructure runs continuously [18].
The ML fallback breaks that. It is a lightweight, always-warm inference stub kept deployed in the same secondary cluster, either a smaller or quantized model or a cached-prediction service [17]. Always-warm means running. The near-zero cluster therefore carries one workload that never scales down, and it is there because of the estate on the other side of the link [17][4].
The post is explicit that the stub does not replace the full on-prem model, and that its job is to keep the application functional but degraded while the primary ML path is restored [17]. How degraded is not stated. There is no accuracy figure for the stub against the production model, and nothing on how a calling service knows which of the two answered [23]. Fifty-plus inference services sit behind that gap [3].
One example is given of the mismatch being designed around: a 15-minute RTO for the web tier against 24 hours for the on-prem ML training pipeline, because that pipeline depends on a single data center's GPU cluster [8]. The post labels this a typical mismatch, so the numbers illustrate someone else's stack [8]. The two targets differ by a factor of 96 [19]. The prescribed remedy is graceful degradation instead of pretending both tiers fail over at the same speed [9].
Why the models are on-prem is also in the text: data residency rules, GPU capex already sunk into a data center, and models trained on sensitive data that cannot leave the corporate network [2]. A second data center answers none of those, and no on-premises failover site appears anywhere in the supplied text [24], so the four rungs of the ladder are choices about the AKS estate [5].
Velero manifests, Helm releases and persistent volumes, managed-disk backups, and Azure SQL or Cosmos DB exports are all objects someone has to restore under pressure [10][20]. One item on the backup list removes a step instead of adding an object: ACR geo-replication, which keeps images durable in a second region with no manual restore step [11]. The model artifacts are on the replication list. Nightly or better replication of trained model binaries, feature-store snapshots and the MLflow registry to Azure Blob Storage is what the post calls "the critical, often-forgotten piece" [12].
No monetary figures appear in the supplied text [22], so "low ongoing cost" is carrying the whole argument. At the cheapest tier the recovery sequence is spelled out: provision a fresh AKS cluster, redeploy workloads from Velero, and stand up an emergency inference service from the last replicated model, with RTO in hours and RPO set by the last backup cycle [14].
What to watch
- Whether the Warm Standby and Multi-Site sections, once available in full, place a second inference site on-prem or keep the fallback in Azure.
- A measured accuracy number for the quantized stub against the full on-prem model; that number sets how long degraded mode is tolerable.
- Whether the always-warm stub's running cost is reported separately from the skeleton cluster's.