Single-pass writes and fewer Step Functions waves drove most of a Glue pipeline's 56% runtime cut

Step Functions Map concurrency caps how many items run at once, and the run ends when the last item finishes ^[3]. At 30 slots, 63 tables need three waves. At 50, they need two ^[1]. The boundary is at 32. Any cap from 32 to 62 gives two waves on tables of similar size, and only 63 gives one ^[1]. The change cut ingestion by roughly 42%, about six minutes ^[2]. "Waves are a cliff, so the payoff is in crossing a boundary, not nudging the number," the author wrote ^[5].

Two conditions decide whether that transfers. Table sizes have to be close enough that wave count predicts wall clock. The author's advice to start the heaviest tables in wave one exists because they are not ^[6]. The Oracle source also has to take 50 concurrent extracts ^[4]. The same post describes an ingestion job whose JDBC partition column is a computed expression Oracle cannot index ^[18]. Every connection full-scans the window to find its slice, and at prod scale those concurrent scans blow the buffer cache and stall ^[18]. A run of 8.9 million rows took 8 minutes; 25 million ran 2h45m and climbing ^[18]. Rows grew 2.8 times and runtime about 21 times ^[7]. The author checked before blaming skew. A bucket-distribution query came back near-uniform, and the skew theory went ^[18]. The fix, partitioning on a prunable column, has not been implemented or measured ^[18].

The largest measured stage saving came from the publish layer ^[3]. Its rename, newline and standardize passes each re-read the whole output on the driver, redoing work the executors had already finished ^[7]. Writing once and parallelizing reconciliation saved 18 minutes, a 72% cut for the stage ^[3]. "No logic changed, just how many times the data got touched," the author wrote ^[9].

The post reports 56% end to end without total minutes, so any split across workstreams is an estimate ^[1]. The measured stage savings add to about 29 minutes, 24 of them from the wave change and the publish rewrite, if the stages run in sequence ^[8].

The outbound job ran on Glue Spark with 6 G.2X DPUs, yet did everything serially on the driver. Spark built one logging DataFrame at the end ^[10]. The compression change bought time with slightly larger zips. The author wrote that tight downstream storage or bandwidth limits would justify a higher level ^[11].

Encryption had a default working against it. GPG compresses before it encrypts, and the input was already zipped, so one batch grew from 5,246 MB to 5,312 MB ^[12]. GPG was spending CPU to add 66 MB ^[4]. The flag `--compress-algo none` fixes it ^[12]. Parallel encryption then OOM-killed the driver. Each multi-GB ZIP was read fully into memory alongside an armored copy, so encryption stays serial ^[13]. The staged fix streams file to file with `gpg.encrypt_file`, gives each call its own GNUPGHOME, and writes unarmored binary about 33% smaller ^[14]. "The blocker was never concurrency, it was the memory model," the author wrote ^[15].

The SFTP job did no distributed work at all ^[16]. At about 11 MB/s, a 5 GB file takes under eight minutes, and more connector concurrency changed nothing because the ceiling was throughput ^[16][5]. The move to Python Shell saves about $52 a year, an 83% cut ^[6].