Salesforce Data 360 injects a Spark listener plugin at its job gateway to find idle executors

The four authors state the premise in one sentence. They wrote, "Successful completion is evidence of a working configuration, not an efficient one." ^[1][7] Their workloads include short runs and also jobs that process terabytes over several hours ^[2]. Averaged across a day, eight million runs is about 93 Spark applications every second ^[18].

The evidence they wanted was per executor. Which executors received tasks, how much CPU and memory each used, and whether execution spilled or retried ^[8]. Those answers let the team separate unused capacity from memory pressure and shuffle bottlenecks ^[3]. If you cut without them, the saving can turn into slower runs, expensive retries or production failures ^[4].

Application teams own the code. Asking each of them to add instrumentation creates an integration project before anyone can look at the waste ^[9]. Data 360 already had a single entry point. The Data Processing Controller, or DPC, is the common gateway for managed Spark submissions behind ingestion, segmentation and activation ^[10]. Spark's listener hooks observe execution independently of application code, so the team extended the open-source DataFlint plugin and had DPC inject it as it built each submission ^[11].

The listeners run on a dedicated listener-bus queue ^[13]. If the instrumentation cannot be configured or initialised, the submission proceeds anyway ^[13]. I think fail-open is the right default for a platform team that does not own the applications. The authors set out to avoid giving those applications another way to fail ^[9], and a collector able to block a submission would be one.

The authors asked whether the 40-executor job needed that capacity immediately, or could start smaller and add executors as demand appeared ^[17]. Their sections on collection and interpretation do not include the answer, or a savings figure.