Eleven terabytes of rasters, no raster database: the two decisions doing the work

DailyMeteo holds about 11 TB of GeoTIFFs, 1 km daily temperature and precipitation for all land on Earth going back to 1961, and there is no raster database anywhere in the system ^[1][2]. According to the project's writeup on dev.to, a Django app reads the files off a disk: no PostGIS raster tables, no tile server in the read path for point queries, no object store ^[2][3]. The interesting part is not the absence. It is which two decisions are actually carrying the weight. The first is workload separation. The archive is produced on one machine and served from another ^[4]. The production box has a terabyte of RAM, most of it in use, and does nothing but run the interpolation ^[5]. The stated reason is blunt: space-time kriging over a continent will consume every core for six hours, and then a user request times out for reasons nobody will connect to weather ^[6]. Everything user-facing lives on the other machine as a Docker Swarm stack: API, frontend, database, GeoServer, workers, reverse proxy ^[7]. The second is the transfer. An rsync runs every 30 minutes ^[8], which is 48 passes a day ^[9], and two properties of it are described as load-bearing. It pulls rather than pushes: the serving machine reaches in and takes what is missing, and the only thing running on the production box on the serving side's behalf is a remote rsync under `nice -n 19 ionice -c3` with a bandwidth cap ^[10]. If the transfer is slow, the interpolation does not care ^[11]. A push would have moved the scheduling decision onto the machine whose entire job is to not be interrupted ^[12]. The second property is that it now transfers incomplete dates. It used to require all 24 rasters for a date, so when one zone failed, five zones of perfectly good output stayed on the production box and the API reported the date as empty ^[13]. With that in place, the read path is string formatting. Files follow the convention `{var}_day_{YYYYMMDD}_equi7{_early|_late|}.tif` ^[14], so a point query for 400 dates means working out the continental zone, building 400 paths, and opening the ones that exist ^[15]. No index to keep in sync, no ingest step, no migration when a year of backfill lands, no second system that can disagree with the disk about what exists ^[16]. When the team needed an authoritative inventory, `os.scandir` over 24 directories returned 580,000 entries in about a second ^[17], an average of roughly 24,000 files per directory ^[18]. The costs are stated rather than glossed. There is no query planner, so a question like "every date where the July mean exceeded X anywhere in Europe" is unanswerable in this shape ^[19]. Directories of 24,000 files are fine on ext4 and fine for `scandir`, but `ls` makes you wait and any tool that stats every entry becomes the bottleneck ^[20]. And the convention is the schema, enforced by nothing: a file with an unexpected suffix is a silent data bug, which is how the system served duplicate values for 3,624 days ^[21], close to ten years of dates ^[22], because a variant appeared that the read path's ranking did not know about. The mitigation offered is not a database but a single function that knows the convention and is the only place it exists ^[23]. Reading raw files is fast enough because the rasters are Cloud Optimized GeoTIFFs: Int16, LZW-compressed, internally tiled in 512x512 blocks, with five overview levels baked in ^[24]. A European tile is 8229 x 5588 pixels and 10.2 MB on disk ^[25], but one pixel costs one block, not 10 MB ^[26]. Int16 instead of a float is a factor of two across 11 TB, about five and a half terabytes of disk ^[27]. Watch the enforcement gap.