Databricks raises Lakebase's 1 GB shared buffers cap to most of the machine's memory

Postgres runs a separate operating system process for each active connection ^[2], and every backend maps shared buffers into its own address space, which requires its own page table entries, the kernel-maintained structures the hardware walks to translate virtual addresses to physical memory ^[3]. Linux does that mapping across 4 KB pages by default ^[4]. One gigabyte of mapped shared buffers therefore costs 262,144 page table entries inside each backend ^[5], and the bill is charged again for every connection you open ^[2]. That is why the Databricks post says sizing shared buffers at 75% of DRAM on fixed-size computes was not as simple as a configuration change ^[17], and why the shipped configuration treats huge pages as load-bearing, not a tuning flourish ^[6].

Up to about 2x throughput is the figure in the summary, and it arrives there without a workload attached to it ^[11]. The conditions for it to transfer are legible from the mechanism. Your working set has to exceed 1 GB, because that was the largest shared buffers size ever configured, and everything above it was served by the slower local file cache tier ^[8][9]. It also has to fit inside the memory now handed to shared buffers, because a miss in both tiers still routes from the compute node to distributed storage ^[10]. A hot set that already fit in a gigabyte was already being served from shared buffers, and this change moves nothing for it.

Databricks turned the new configuration on at CU 80 and above; Neon at CU 18 and above ^[7], a gap of roughly 4.4x between the two floors ^[20]. Whether that reflects different memory per CU or different appetite for risk on each platform is not something the published thresholds settle.

The two descriptions of what the local file cache actually is do not line up. In the body of the post, the LFC takes whatever remains of a compute cache budget of up to 75% of DRAM once shared buffers have their gigabyte ^[8]. In the Neon summary, the same pages were falling through to a local disk cache ^[12]. Reads in a disaggregated system such as Lakebase do not travel through the OS filesystem or its page cache ^[22], which makes the memory accounting for a file-backed cache tier the interesting gap. How much of the 2x is DRAM against disk, and how much is one tier against another, depends on that answer.

Two of the wins follow from structure alone, with no benchmark behind them. Double buffering goes away, so a gigabyte of cached pages consumes a gigabyte of RAM instead of two ^[15][21]. And because the cache lives inside Postgres rather than the kernel, eviction can be decided with knowledge of database state, which the post frames as future positioning to pursue smarter replacement policies ^[16]. The authors call the LFC a clever and pragmatic solution and say they intend to retire its current form as they move to fully dynamic shared buffers ^[13][14]. It is still in use on all compute ^[13], so until part 2 the autoscaling path is the old path.