A KAYAK scraper that quit polling early priced 18 of 24 itineraries above a later run
A KAYAK scraper's first run quoted higher fares than a run three minutes later on 18 of 24 itineraries, its author reports. The poll returns the cheapest fare among sellers who have answered so far, so a count-based stop gives a high price.
The Engineer · Build desk
Key figures: in the first run, which stopped at 5 seconds, KAYAK reported 29 itineraries. In the later run, at 15 seconds, it reported 488. Across 24 itineraries both runs shared, the median fare gap was 7.1% and the largest was 43.8%.
| Figure | Value | Note | Claim |
|---|---|---|---|
| KAYAK itineraries found, first run at 5 s | 29 itineraries | 7 | |
| Found, later run at 15 s | 488 itineraries | 7 | |
| Median fare gap, 24 shared itineraries | 7.1% | largest gap: 43.8% | 4 |
What happened
- Across those 24 shared itineraries the median gap was 7.1% and the largest was 43.8%, according to the scraper's author.
- The run log shows the earlier run stopping at 5 seconds with 29 itineraries found, while the run 180 seconds later reached 15 seconds and 488.
- The author traced the early stop to his own wait loop, which breaks once the flights seen reach the buyer's row cap.
Why it matters
- constraint In this scraper the row cap doubled as a price-completeness setting, so a buyer who asked for fewer rows got dearer fares with nothing in the input to warn them.
- exposure Cheapest-seller or seller-share findings drawn from early reads would report which sellers answer fastest, and could be published as a market finding.
- decision A pipeline needs a stop rule that is not a row count or an itinerary total. The author calls the per-row providersForThisFlight field the confidence signal.
KAYAK does not return a flight list in one response. The results page opens a search and polls it through POST /i/api/search/dynamic/flights/poll, and each answer carries results whose bookingOptions list one entry per seller quoting that itinerary [5]. Sellers do not all answer at once. According to the scraper's author, an early answer carries one or two options per itinerary and a later one carries thirty [6]. The price field is the minimum over whatever has arrived, so an early read is too high every time [6].
The trigger was in the author's own wait loop: `if (n >= input.maxResults) break;`. The buyer asked for 30 rows, 36 itineraries had arrived, and the loop stopped [9]. The author wrote: "That is correct code over an incomplete input." [10] The answer that first clears a row cap is also the thinnest on prices [9].
The search's own totalCount is a snapshot too, so it is no stop signal. The log shows 29 itineraries found at 5 seconds and 488 at 15, a ratio of about 16.8 [c7, d2].
The cap test isolates the variable. Both runs started in the same second from one account on 4 GB, with identical input except a maxResults of 10 and 50 [11]. The 50-row run is the better one but looks unfinished. The ten shared itineraries had a median of 8 sellers there, under the 10-plus the author counts as full [c12, c14]. The $57 Jetstar fare, the cheapest way to fly Bali to Singapore that day, was missing from the ten-row run because Jetstar had not answered [13].
Seller attribution skews as well. Kiwi.com is the named cheapest seller on 45 of 55 thin rows, about 82%, and on none of the 80 full ones [c14, d3]. "Kiwi is not cheap here; Kiwi is fast," the author wrote [15].
Treat all of this as one author's measurement. It covers one route, one departure date and one Actor, and the seller counts come from six runs [c2, c14]. The 43.8% is the largest of 24 gaps, against a median of 7.1% [4]. The 24 are itineraries both runs returned, so fares absent from the early read, like the Jetstar one, are outside the 18-of-24 count [c3, c13]. For that 75% to transfer to another pipeline, its sellers would have to answer as unevenly as they did on Bali to Singapore, and its stop rule would have to fire as early as a 36-flight count did [d1, c7].
What to watch
- A re-run of the same DPS-SIN comparison with the row-cap exit removed would show whether a 15-second read still sits above a settled price.
- Whether the Kiwi.com skew on thin rows repeats on routes other than Bali to Singapore.
- Whether the author's Actor changes its default stop rule or filters rows by providersForThisFlight.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence62
- Adoption
- Insufficient
- Hype gap−5
- Incentives
- Insufficient
- Confidence58
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
The author runs an Apify Actor that reads KAYAK flight searches into rows with price, seller (airline or agency), flight numbers, times, stops, layovers, aircraft and a booking link.
- [2]
The test search was DPS to SIN, departing 15 March 2027, one way, economy, sorted by KAYAK's 'best', run on 13 and 14 October, two of the runs three minutes apart.
- [3]
Across the 24 itineraries the 13 October pair had in common, the second run was cheaper on 18, identical on 6, and dearer on none.
- [4]
Across those 24 itineraries the median gap was 7.1% and the biggest was 43.8%.
- [5]
KAYAK's results page does not fetch a list of flights; it opens a search and then polls it via POST /i/api/search/dynamic/flights/poll. Each answer carries results[], and each result carries bookingOptions[], one entry per seller quoting that itinerary with its own price and deep link.
- [6]
Sellers do not all answer KAYAK at once: an early poll answer carries one or two bookingOptions per itinerary and a later one carries thirty. The price is a minimum over whatever has arrived, and the minimum of a growing set, taken early, is too high every time, according to the author.
- [7]
The run log of the two 13 October runs, 180 seconds apart, shows the first run with 36 flights in 5 s and 'KAYAK found 29 itineraries', and the second with 63 flights in 15 s and 'KAYAK found 488 itineraries'.
- [8]
The search's own totalCount is a snapshot too, read five seconds in instead of fifteen.
- [9]
The author's wait loop contained the line 'if (n >= input.maxResults) break;' where n is flights seen so far. The buyer asked for 30 rows, 36 itineraries had arrived, so it stopped; the poll answer that first clears the row cap is also the thinnest on prices. The row cap had quietly become the price-completeness setting, and nothing in the input hinted at that.
- [10]
That is correct code over an incomplete input.
- [11]
Two runs were started in the same second from one account, both on 4 GB, with identical input except maxResults, set to 10 in one and 50 in the other.
- [12]
Ten itineraries appear in both the 10-row and 50-row runs. Six are cheaper in the 50-row run, four are identical, none is dearer. Median seller count was 4 in the ten-row run against 8 in the 50-row run.
- [13]
The $57 Jetstar fare, the actual cheapest way to fly Bali to Singapore that day, is absent from the ten-row run because Jetstar had not answered yet.
- [14]
Across all six runs, 55 rows had only 1-2 sellers reported and 80 rows had 10 or more (the author calls the latter 'the full ones'). Kiwi.com was named the cheapest seller in 45 of the 55 thin rows and in none of the 80 full ones.
- [15]
Kiwi is not cheap here; Kiwi is fast.
- [16]
providersForThisFlight ships on every row; the author calls it the confidence field and says he had been treating it as trivia.
- [17]
18 of 24 shared itineraries were cheaper in the later run, which is 75%.
- [18]
The itinerary count KAYAK reported at 15 seconds was about 16.8 times the count at 5 seconds.
- [19]
Kiwi.com was named cheapest on about 82% of thin rows.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.toAsking KAYAK for 10 Flights Instead of 50 Made the Same Seat 30% Dearer
1 article · October 13, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- Web ScrapingFollow
- Flight metasearchFollow
- Data QualityFollow