BuildNot yet confirmed elsewhere1 publisher2 min readPublished
Traefik's 3.7 announcement nests service middlewares one level too deep for the parser
Traefik 3.7.14 rejected the announcement post's service-middleware YAML with "field not found, node: middlewares", a dev.to author reported. The key belongs on the service beside loadBalancer. Moved there, it kept a canary-only header off stable traffic in every test, so configs copied from the post need a parse check first.
The Engineer · Build desk
One author's tests: a 200-request sequential run matched the 9:1 split, implying 20 canary and 180 stable requests; X-Canary hit every canary request and no stable one. A 500-request run fired 50 at a time showed no leaks either way.
| Figure | Value | Note | Claim |
|---|---|---|---|
| Sequential run, split matched 9:1 | 200 requests | 9 | |
| Canary requests implied at 9:1 | 20 requests | 180 stable requests implied | 15 |
| Concurrent run, no leaks | 500 requests | fired 50 at a time | 10 |
What happened
- The announcement's example nests the middlewares key inside loadBalancer, beside servers.
- The author checked the Go source for 3.7.14: the Service struct has a Middlewares []string field and ServersLoadBalancer has no such field.
- In a 9:1 weighted setup, the X-Canary header landed on every canary request and on none of the stable ones across 200 sequential requests.
- When a router middleware and a service middleware both set X-Trace, the service value won on all 500 canary-branch requests.
Why it matters
- cost Anyone pasting the post's snippet into a file-provider config hits a build error that names the key but gives no hint of where it belongs.
- decision Teams that layer router and service middlewares now have to decide which level owns each header, because the service value overwrites the router's on the canary branch.
- constraint Hand-testing a canary with a handful of requests can miss the canary slot entirely, because the split follows a fixed cycle.
Before 3.7, a middleware could attach only to a router or run at the entry point, so it applied before Traefik had picked a backend [2]. Whatever you attached ran for every request through that router, whichever child of a weighted service ended up serving it [2]. In 3.7 a middleware can attach to a service, including one child of a composite service, and it runs after the load balancer has chosen a backend [3].
The error is easy to hit. The author's container logged it on every restart, using the file provider [5]. The author's verdict on the cause: "That's not a typo on my side." [14]
The test design is sound. After 200 sequential requests, the author fired 500 requests with 50 in flight at once through xargs -P 50, so that any race between the load balancer choosing a branch and the middleware chain running would show up [10]. At 9:1 weights, a clean run means 20 canary requests out of 200 and 50 out of 500 [15] [16]. The author reports the same split and no leaks in either direction on the concurrent pass [10].
The middleware under test only set a header, and the backends were two small Python servers that echo what they receive [8]. A middleware that holds state or rejects requests has to behave the same way inside the branch chain, and the tests covered only a header setter. They also ran on the file provider only [5]. The write-up does not report whether Traefik has corrected the post.
In our view, the Go struct is the spec for this feature and the post's snippet is a sketch of it [6] [4]. The feature itself held up in the author's runs.
What to watch
- Whether Traefik corrects the example in the 3.7 announcement or its service-middleware docs.
- Test reports from providers other than the file provider, since the write-up used only the file provider.
- Reports on middleware that rejects requests or holds state on a weighted child; the tests here covered header setters only.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence72
- Adoption
- Insufficient
- Hype gap−10
- Incentives
- Insufficient
- Confidence65
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Service middlewares are the headline feature of Traefik 3.7 ("Langres").
- [2]
Before 3.7, a middleware could attach only to a router or run at the entry point, so it applied before Traefik picked which backend would serve the request; on a weighted service, whatever was attached ran for every request through the router regardless of which child handled it.
- [3]
Traefik 3.7 lets a middleware attach directly to a service, including one child of a composite service, so it runs after the load balancer has already chosen a backend.
- [4]
Traefik's announcement post shows the service-middleware example with the middlewares key inside loadBalancer, next to servers.
- [5]
The author pointed a real Traefik 3.7.14 container at a minimal version of the post's example using the file provider and got the error "field not found, node: middlewares" on every restart, logged as "Error while building configuration (for the first time)".
- [6]
In the Go source for that version, the Service struct has a field Middlewares []string and the ServersLoadBalancer struct does not have that field at all.
- [7]
Moving middlewares up one level, out of loadBalancer and onto the service as a sibling of loadBalancer, was the whole fix; the same container then started cleanly.
- [8]
With the syntax fixed, the author built a weighted service with a 9:1 split between stable-svc and canary-svc, each a small Python server echoing back received headers, with a canary-header middleware on canary-svc that sets X-Canary: true, and fired 200 sequential requests through the router.
- [9]
In the 200-request run the split matched the 9:1 weight exactly, and the X-Canary header landed on every canary request and on none of the stable ones.
- [10]
The author repeated the test with 500 requests fired 50 at a time with xargs -P 50, to rule out a race between the load balancer's branch selection and the middleware chain under concurrency, and saw the same split, the same isolation, and no leaks in either direction.
- [11]
The weighting is deterministic, not random sampling: roughly one canary request in every ten in a fixed pattern, so a hand test with a handful of requests might never hit the canary, or might mistake a fixed slot in a cycle for a probability.
- [12]
The announcement says router middlewares run before service middlewares; the author checked this with a router-level middleware setting X-Trace: router and a service-level middleware on the canary branch setting X-Trace: service.
- [13]
The service middleware overwrote the router middleware's X-Trace header on every one of the 500 requests on the canary branch, so the backend saw service.
- [14]
"That's not a typo on my side."
- [15]
A 9:1 split over 200 requests implies 20 canary requests and 180 stable requests.
- [16]
A 9:1 split over 500 requests implies 50 canary requests and 450 stable requests.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.toTraefik 3.7's service middlewares target one branch of a weighted service
1 article · October 12, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.