Build1 publisher2 min readPublished
Mapping a green Karate suite onto its OpenAPI contract leaves half the operations unproven
A dev.to walkthrough maps a fully green Karate suite back to its OpenAPI paths and finds half the customer operations with no scenario behind them, including both destructive methods on the resource.
The Engineer · Build desk
What happened
- In the post's worked example, CI reports 184 Karate scenarios, 184 passed, zero failed, a 100 percent pass rate.
- Set against the current OpenAPI contract, that suite covers GET and POST /customers plus GET /customers/{customerId}, while PATCH, DELETE and the orders sub-path have no mapped scenario.
- The proposed unit of coverage is an OpenAPI operation, meaning HTTP method plus normalized path, matched against Karate requests by a deterministic comparison.
- Each operation keeps its own evidence record: feature file and scenario name when covered, an empty evidence array when not, so the figure can be inspected.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Priority for new tests has to come from the uncovered operation list, because execution health only counts scenarios that already exist.
- decision Anyone adopting this has to choose the default for ambiguous matches, and the conservative choice means the tool will under-report coverage on suites that assemble URLs in code.
- exposure The gaps in the example cluster on the destructive methods, so the calls that change or remove a customer record are the ones running without test evidence.
- precedent Once operations are enumerated from the contract, every operation added to the spec arrives uncovered by default. A contract change becomes a visible test debt item on the next run.
The join between a Karate suite and an OpenAPI file is string handling. Karate lets the same request be written as `path 'orders', orderId`, as `path 'orders/' + orderId`, or as `url baseUrl + '/orders/' + orderId`, while the contract writes it once as `/orders/{orderId}` [7]. Literal comparison misses all three [8]. The dev.to post normalizes both sides into segment lists instead: the contract becomes `[orders, {param}]`, the test becomes `[orders, <dynamic>]`, and the methods compare directly. A match rests on structural compatibility instead of source spelling [9].
That comparison is cheap to run. "You do not need an LLM for this basic comparison. It should be deterministic and explainable," the author wrote [10]. The design decision worth arguing about is the default when segments do not resolve. "A false covered result is more dangerous than an unknown result because it hides work," the post says [11]. Under that rule a request whose path is assembled at runtime lands in unknown, not covered, so a suite full of concatenated URLs reports less coverage than it has [3]. I would take that trade.
The example's numbers are worth doing by hand. Three of the six customer operations listed have a mapped scenario and three do not, which is 50 percent contract coverage under a 100 percent pass rate [1]. Two of the three gaps are the destructive methods on `/customers/{customerId}`, PATCH and DELETE, and the third is an entire sub-resource, `GET /customers/{customerId}/orders` [2].
Those figures are a constructed illustration, not a measurement of anyone's real suite. Two conditions have to hold before the ratio means anything on yours. The OpenAPI file has to describe the API you actually ship. The operation list is derived from the contract, so an operation the contract omits is not reported as uncovered at all [4]. And enough of your requests have to resolve statically for the normalizer to see them, or the conservative default fills the report with unknowns.
The post's argument for operations over application code coverage is narrow and, I think, correct. Method plus normalized path answers whether the externally documented surface was exercised, which line coverage does not [15].
The post is explicit about the limits of this measure. The scenario `Given path 'orders', orderId / When method get / Then status 200` counts as evidence for `GET /orders/{orderId}`. Finding that mapped scenario proves the operation was exercised without proving the scenario is strong; the author points to contract-relevant assertions as the stronger form [13].
What to watch
- Whether teams wire the uncovered-operation list into CI as a build gate, so a new contract operation fails the pipeline until a scenario exists.
- Whether the mapper handles OpenAPI server prefixes and operations that differ only by query parameters, which segment normalization on its own will not separate.
- Whether coverage reporting grows a second axis for assertion depth, given that a status 200 check counts as full evidence today.