Build1 publisher3 min readPublished
An 8-minute dashboard heartbeat held a Databricks serverless warehouse open all weekend
A dev.to account puts one weekend of Databricks serverless compute at $14,000 and traces it to a Tableau connection that polled information_schema every eight minutes, restarting a 10-minute auto-stop timer it never let expire.
The Engineer · Build desk
What happened
- A dev.to account describes a billing alert at 2:14 AM on a Sunday reporting that the team had used 80 percent of its monthly cloud spend in 48 hours.
- The warehouse was sql_warehouse_prod_v2, a Serverless SQL Warehouse sized Large, which the author says runs at a significantly higher DBU rate than a heartbeat ping needed.
- A Tableau connection hit the warehouse with a system.information_schema query every 8 minutes, and the 10-minute Auto-stop timer reset each time without ever reaching its idle threshold.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Any client that polls faster than the timeout keeps auto-stop from working as a cost control, so the levers that actually bound spend are the client's keepalive interval and the warehouse size.
- decision Anyone pointing a BI tool at a serverless warehouse now has to choose the timeout against that tool's poll interval, and decide whether dashboards get their own small warehouse.
- exposure A billing alert as the last line of defence reports money already committed. Tags identify which warehouse spent it and leave its next weekend uncapped.
- contradiction The account's headline dollar figure and its 80-percent alert cover different scopes, so the failure mode is legible and the rate a reader would pay for it is unverifiable.
Auto-stop measures idleness. A serverless SQL warehouse counts down only while no query arrives. Two numbers therefore decide the bill, and they are set in different places: the timeout in the warehouse config, and the keepalive interval in the BI client. In the dev.to account the timeout was 10 minutes and a Tableau connection queried system.information_schema every 8 minutes, so the timer restarted before it could expire [9]. Across a 48-hour weekend that is about 360 restarts [1].
The pings were sub-millisecond and did not appear in the team's main performance monitoring [10]. Before finding them, the author checked the cluster logs for DBR 13.3 LTS and spark_query_history, and saw the standard daily ingestion patterns [5]. A new pipeline had shipped on the Friday, green in CI, with data landing on time [4]. The post puts the spend on the interaction between the BI tool and the warehouse, kept alive by a service principal with broad CAN USE permissions [7][8].
The two money figures in the post cover different scopes. The headline puts the weekend at $14,000 [2]. The alert reported 80 percent of monthly cloud spend consumed in 48 hours [1]. Read as the same money, that implies a monthly budget near $17,500 and a burn of roughly $292 an hour [2][3]. The post does not state the DBU rate for the Large size or publish the billing export [18]. Without either, the heartbeat warehouse's share of the total stays hidden.
Auto-stop bounds spend only when it is set shorter than the client's poll interval [5]. The author severed the BI connection and moved the setting to 1 minute [11]. At 1 minute against an 8-minute heartbeat, the warehouse is up about one minute in eight, near a 12 percent duty cycle, paid for with a cold start on each resume [4]. On the client side, the connection now points at a specific Unity Catalog schema that does not need the broad information_schema scan [13]. Dashboards went to a Serverless-Small warehouse and the Large one stayed for ad-hoc analyst queries and heavy ELT [12].
The standing controls apply at review time. No production warehouse may exceed Medium without a documented exemption in the Terraform repo, and a PR for anything larger triggers a CI cost-estimate check against the Databricks Billing API to flag the daily run-rate [15]. The gate checks size. The warehouse here was already provisioned and the change came from a client session, so the gate had nothing to fire on [6].
The tags are for attribution. "While this didn't stop the spending, it allowed me to isolate the DBU consumption at the warehouse level in the billing export CSVs within minutes, rather than waiting for the bill to aggregate," the author wrote of adding CostCenter and Owner to the warehouse [14].
Four conditions have to hold together for this number to reach your setup: a client that keepalives on a fixed interval, credentials that keep the session open, a warehouse sized above what the polling needs, and an auto-stop longer than the poll interval [3][8][9]. The account notes that serverless is designed to stay available, and that a traditional cluster would eventually have hit a resource contention limit or a timeout [17]. "Serverless means you don't manage the nodes, but you absolutely have to manage the session lifecycle," the author wrote [16].
What to watch
- Any Databricks control that halts a serverless SQL warehouse at a DBU ceiling instead of alerting on one.
- Whether Tableau and comparable BI clients expose the keepalive interval per Databricks connection, and what the default is.
- A published billing export or DBU rate that would let the weekend total be split by warehouse.