Build1 publisher2 min readPublished
FrugalGPT fits a fresh triage rule for every dataset and task it is tested on
The paper reports matching GPT-4 with up to 98 percent less spend. The saving rests on a 150x spread in prompt prices between providers and on a triage rule fitted separately per dataset and task.
The Engineer · Build desk
What happened
- The FrugalGPT paper compares the cost of 12 commercial LLMs from providers including OpenAI, AI21, CoHere and Textsynth, and finds fees that differ by two orders of magnitude.
- Its price table puts the prompt cost for 10M tokens at $30 on OpenAI's GPT-4 and $0.20 on GPT-J hosted by Textsynth.
- The authors set out three cost-reduction strategies, namely prompt adaptation, LLM approximation and LLM cascade, and implement the cascade as FrugalGPT.
- The reported experiments match the best single model, given as GPT-4, with up to 98 percent cost reduction, or beat its accuracy by 4 percent at the same spend.
- For each dataset and task, FrugalGPT learns how to triage queries to different combinations of ChatGPT, GPT-3 and GPT-4.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Trying a cascade is cheap to get wrong: at the paper's prices, a rejected GPT-J call adds 0.67 percent to the GPT-4 prompt cost behind it, so the downside of a bad route is small next to the upside of a good one.
- constraint Because the triage is learned per dataset and task, every new workload needs its own labelled examples and its own fit before the reported saving means anything for that workload.
- decision Teams get an operating point to choose: the same method is offered as equal quality at a fiftieth of the spend or as 4 percent more accuracy at the current spend.
- capability Fitting a cascade across vendors leaves you with a second provider already wired up and prompted, which is the fallback the authors say a single-provider setup lacks when demand spikes.
The escalation rule decides whether any of this transfers, and the supplied text does not give it. The paper describes LLM cascade as focusing on "how to adaptively choose which LLM APIs to use for different queries" [9], and FrugalGPT as a cascade that "learns which combinations of LLMs to use for different queries" [5]. What score the first answer has to miss before the second call fires is not in the material. The authors do say why that part is hard: earlier cascade work, including FrugalML, was built for predictive tasks with a known set of labels, and does not account for the full capabilities of an LLM [14].
Up to 98 percent off means paying about a fiftieth of the single-model bill [1]. The price spread measured in the paper is wider than that, at $30 against $0.20 for 10M prompt tokens, a factor of 150 [3][2]. A cascade cannot capture the whole 150x. A router that did would be sending every query to the cheap tier.
The cheap tier costs something every time it is wrong. An escalated query pays for both calls, and at those prompt prices the discarded first call costs 0.67 percent of the GPT-4 prompt cost it precedes [3]. Generation cost and any fixed per-query fee sit on top of that [8], so the true overhead depends on how long the rejected cheap answer ran.
The 98 percent transfers only if your traffic looks like the paper's. Most queries have to be ones the cheap model gets right, and the paper's own comparison cuts against that for hard work: aggregated GPT-J responses reach GPT-3-like performance on relatively simple tasks, while GPT-J can be much worse on difficult ones [10]. You also need labelled examples for each new task, because the triage is learned per dataset and task, across ChatGPT, GPT-3 and GPT-4 [7]. And the price gap you are arbitraging has to be the one in Table 1, which lists 12 commercial models from OpenAI, AI21, CoHere and Textsynth [2].
The framing costs are cited, not measured: over $700,000 a day to operate ChatGPT [12], and over $21,000 a month for a small business using GPT-4 for customer service [13]. Both come from references rather than the authors' own instrumentation.
Cost is not the only reason the paper gives for spreading queries across vendors. It argues that relying on one API provider is not reliable if that provider becomes unavailable, potentially due to spiking demand [11]. And the same experiments report a second operating point at the other end of the curve: 4 percent better accuracy than GPT-4 for the same spend [6].
What to watch
- Whether a later version of the paper publishes the scoring function and threshold that decide when a query escalates.
- Whether the $30 against $0.20 prompt price spread in Table 1 still holds at current provider list prices, since the ceiling on the saving moves with it.
- Cascade results on tasks with no labelled training split, which would test whether the per-dataset fit is required.