Skip to content

Build1 publisher2 min readPublished

Fanning 14,000 accounts across 500 Lambdas keeps BMW's daily cost forecast near $50 a month

BMW Group's CLEA forecasts hundreds of thousands of account-service pairs every day with Prophet on Lambda, then has to decide which of those deviations is worth an account owner's email. Compute runs about $50 a month.

The Engineer · Build desk

Illustration accompanying Fanning 14,000 accounts across 500 Lambdas keeps BMW's daily cost forecast near $50 a month

What happened

  • BMW Group's CLEA, which started as QuickSight dashboards over cloud spend, now runs anomaly detection every day and emails account owners when spending departs from its expected pattern.
  • Every account contributes one time series per active service, so across more than 14,000 accounts the system carries hundreds of thousands of account-service pairs, each with its own forecast and evaluation.
  • Step Functions fans the daily run out through a Distributed Map across as many as 500 concurrent Lambda functions, one account per function, and the full pass finishes in about 20 minutes.
  • Billing data arrives on a one-day lag and the pipeline is scheduled only after AWS Cost and Usage Report delivery is confirmed complete, to avoid analysing partial days.
  • AWS puts the compute cost of processing every account daily at about $50 per month.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint The same adaptive baseline that stops deliberate growth from paging an owner will eventually take a sustained ramp into its own expectation, so gradual overspend is the failure mode this design buys.
  • exposure The earliest an owner can learn about a runaway is the morning after it started, because the run waits for confirmed billing delivery before it forecasts anything.
  • cost At roughly $1.67 a run, what a team copying this pattern actually pays for is engineering time on filter design and alert routing.
  • capability Because the module boundary is two data formats, a team can replace Prophet with something else without regression-testing the detection and alerting path owners read every day.

A fixed dollar threshold is the design everyone tries first, and the AWS post, co-written with BMW Group's Philipp Karg and Data Reply's Christopher Masurek, is specific about why it fails at this size [18]. Accounts grow, adopt new services and ramp workloads on purpose, and a fixed rule treats all of it as anomalous [13]. Set the number high enough for the largest accounts to stay quiet and the smaller ones get no coverage at all [13]. So each account-service pair is measured against its own learned trajectory instead of a number chosen centrally [14].

The authors put the cost of that choice in one sentence: a model that adapts to a trend will eventually absorb one [15]. The text stops mid-sentence at that point, before any filter criteria [22]. Getting from a forecast to an alert needs a baseline, a daily comparison that flags the days falling outside it, and filters that decide which of those days are worth an owner's attention [16]. Impact is defined as actual spend minus expected spend, so a positive number means overspend against the forecast [17].

Fourteen thousand accounts over 500 concurrent slots is 28 accounts per slot, and 20 minutes of wall clock divided by 28 leaves roughly 43 seconds per account if every slot stays busy for the whole run [19]. In that window each function fits Prophet on 365 days of daily history for every service the account uses, with additive seasonality [9]. The account list itself is produced by a preparation Lambda that writes JSON to S3 for the Distributed Map to read [10].

The compute figure divides the same way. At 30 runs a month, about $50 works out to roughly $1.67 per pass over the estate [20], or about a third of a cent per account per month [21]. Prophet was chosen for its simplicity and its steady performance on cost time series, not for accuracy claims [9].

The part I would steal outright is the module boundary. Forecasting is held in place by two formats: daily cost per account-service going in, per-day predicted values with confidence intervals coming out, so the engine can be replaced without touching the detection and alerting layers owners depend on daily [12]. A replacement therefore has to emit intervals, not just point predictions. Those predictions feed two consumers, a 12-month rolling forecast and the per-day expected baseline that detection reads [11]. Upstream of all of it sit roughly 3 billion billing rows across 500 columns a month, aggregated to one grain of daily cost per account per service [6].

What to watch

  • Whether BMW publishes the filter criteria, including any minimum impact or confidence-band rule that decides which flagged days generate an email.
  • Whether the forecasting module is ever actually swapped, which would test the two-format interface under load.
  • How the daily schedule behaves when a provider's billing export is late, given that the run waits for delivery to be confirmed complete.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories