Build1 publisher3 min readPublished
The First Firewall Rule Is a Cutover: One Allowlist Entry, One Dead Production App
A team's CI backup job allowlisted its runner IP on DigitalOcean managed Postgres. The database switched from credentials-only to allowlist-only, and production fell off the list it was never on.
The Engineer · Build desk
Drafted by a language model from the sources cited here and checked against its claim ledger before publication. How we use AISend a correction
What happened
- The team's DigitalOcean managed Postgres had no trusted-source rules at all, which by default means open to anything that has the connection string and credentials.
- The droplet running their application talked to the database from a public IP, with no allowlist entry required.
- Their backup pipeline temporarily allowlists the runner IP, runs pg_dump, then revokes the IP; the author describes this as the standard pattern used by basically every CI-driven DB job.
- The new pipeline added one rule with the command: doctl databases firewalls add "$DB_ID" --rule "ip_addr:${RUNNER_IP}".
- DigitalOcean flipped the database into enforcement mode the moment a single rule was added, moving it from 'anyone with credentials' to 'only IPs on the allowlist'.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
A team added a database backup pipeline that temporarily allowlists the CI runner's IP, runs `pg_dump`, then revokes the IP, which is the standard pattern for CI-driven database jobs [3]. The first run did all three steps correctly and took production down while doing it [4][6].
The setup before the pipeline is the part worth reading twice. Their DigitalOcean managed Postgres had no trusted-source rules at all, which in practice means open to anything holding the connection string and credentials [1]. The application droplet reached the database over a public IP and had never needed an allowlist entry [2]. Then the pipeline ran one command, `doctl databases firewalls add` with the runner's IP [4].
According to the team's account, DigitalOcean flipped the database into enforcement mode the moment that single rule existed [5]. The runner was on the list. The droplet was not, because there had never been a list [6]. The dump succeeded and the application returned 500s for every request for the duration of the window [7]. Traffic was dropped at the network layer, not the auth layer, so from the application's side the database had simply vanished [11].
The semantics are documented, after a fashion. The team eventually found a docs page saying trusted sources are enforced "when configured," where "configured" turns out to mean "any rule exists" [8]. There is no dashboard warning and no confirmation prompt at the transition [9]. That is the whole trap: a zero-to-one rule change is a security-model cutover, while every one-to-many change after it is merely additive [17]. The command looks identical in both cases.
Recovery is cheap once you know. Removing all rules returns the database to open mode [10], and adding the production droplet's IP permanently means later runner entries are just extra rows in an existing allowlist with no enforcement flip [12]. The team also switched from `add` to `append`, which has the same effect for a single rule but makes explicit that you are not overwriting the whole firewall [13].
The durable fix is a preflight, not a runbook note. Their workflows now resolve the production droplet's public IP, check it against `doctl databases firewalls list`, and exit non-zero with the exact `append` command if it is missing, so an operator has to add the rule once, deliberately, before any workflow can open a runner IP [14]. The same guard went into the backup, seed and restore workflows, everywhere trusted sources get mutated [15].
Worth checking on your own estate: whether any managed database is currently running with an empty trusted-source list, because that is the configuration where an innocuous-looking firewall call becomes an outage. This is one team's report, published on dev.to and originally on their own blog [16], so treat the exact enforcement behaviour as something to verify on a staging database before you rely on it either way.