Skip to content

Build1 publisher2 min readPublished Updated

Dropping Cloudflare from a Prometheus scrape retired two of its three defense layers

A team sent internal metrics out through the public edge before a routine doctl listing showed both droplets on the same private /16. The rewrite keeps the OAuth grant and retires the two route-based controls.

The Engineer · Build desk

Illustration accompanying Dropping Cloudflare from a Prometheus scrape retired two of its three defense layers

What happened

  • Prometheus on one DigitalOcean droplet scraped a Spring Boot actuator endpoint on a second droplet by routing every request out through Cloudflare's edge, the cloud firewall and nginx.
  • The job ran every 30 seconds and moved about 80 KB of metrics on each pass between the API droplet and the observability droplet.
  • A doctl listing run for unrelated reasons showed the two droplets at 10.108.0.3 and 10.108.0.5 in nyc3, inside DigitalOcean's default 10.108.0.0/16 VPC the whole time.
  • The rewritten job connects straight to 10.108.0.3:8080 over the VPC with Cloudflare, nginx and TLS off the path, and a firewall rule allowing tcp/8080 only from tag:impress.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • cost Internal telemetry was paying the full public-request price twice a minute: a handshake with Cloudflare, a second with nginx, a WAF evaluation and a token mint, all charged to the two droplets already sharing a private segment.
  • decision Anyone copying this has to establish first whether the actuator gate is enforced in the app or by the upstream proxy, because deleting the public hop deletes any control that lived in the proxy.
  • exposure With the token URL on http, the client secret goes out and the bearer token comes back in cleartext, so the trust boundary is now whoever can open tcp/8080 inside 10.108.0.0/16.

The check that made the swap safe was one command. From inside jo4-impress, a request to http://10.108.0.3:8080/actuator/prometheus carrying no bearer token came back HTTP/1.1 401 Unauthorized with a WWW-Authenticate: Bearer header [13]. Spring Security's filter chain runs the same way for traffic arriving on the private interface, so the OAuth requirement belongs to the endpoint and not to the network path [14]. The author said hand-verifying it was worth the trouble because being fairly sure on that question is the kind of thing that gets you fired [18].

That distinction decides which of the three controls the move made redundant. A WAF allowlist on /actuator/* is a statement about a route [3]. So is a cloud firewall admitting port 443 only from Cloudflare's published CIDR ranges [5]. The client_credentials grant with scope metrics:read is a statement about an endpoint [4], and the corrected topology keeps it, alongside a single DigitalOcean firewall rule allowing tcp/8080 from tag:impress [11]. The post's own count is three layers of defense with two of them unnecessary [19], and it does not say whether the WAF rule was deleted or just left in place.

A 30-second interval is 2,880 scrapes a day [20]. Each one opened a TLS session with Cloudflare and a second with nginx, then evaluated the WAF rule and minted or reused a token [7], so the pair was running roughly 5,760 handshakes a day [21]. The payload was about 80 KB a scrape [6], near 230 MB a day [22], between droplets at 10.108.0.3 and 10.108.0.5 [8].

Nothing on the new path uses TLS. The job's scheme is http and the token URL is http://10.108.0.3:8080/oauth/token, since the token endpoint lives on the same Spring Boot app [15][16]. According to the post, droplets in one VPC reach each other on private interfaces with no public exposure and no encryption-in-transit requirement, the VPC being a private L2 segment [10]. That segment is 10.108.0.0/16, DigitalOcean's default VPC for nyc3 [9].

This transfers to another stack only under conditions the post happens to satisfy. Both ends have to sit in the same VPC and region [8]. The app has to be listening on a port the private interface can reach, which here it already was, because :8080 served the nginx upstream [12]. And the gate has to be in the app. Test it with a request that carries no credential [13]. The author wrote: "I had built a public path to talk between two machines on the same private network. That's the kind of thing you laugh at in someone else's design review." [17]

What to watch

  • Whether the link stays on DigitalOcean's default nyc3 VPC or moves to a VPC created for these two droplets.
  • Whether the tag:impress rule on tcp/8080 remains the only network control as more droplets join the account.
  • Whether the Cloudflare WAF rule on /actuator/* is removed now that the scrape no longer traverses the edge.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories