Build1 publisher2 min readPublished
A five-column CSV per institution collapses 107 official rate publishers into one dataset
The AllRates dataset on Hugging Face holds 14.6 million rows in each publisher's own direction and label, so the collecting is finished while the inversion, rounding and missing-day rules stay in your code.
The Engineer · Build desk
What happened
- A public Hugging Face dataset now holds 14.6 million official exchange rates collected from 107 central banks and tax authorities, with one CSV per institution.
- Every institution's file uses the same five columns: date, base, quote, type and value.
- Rates are stored in the publisher's own direction, and the maintainers neither flip them nor compute crosses.
- Depth varies by institution: the Swiss National Bank series starts in 1914 and the Bank of Korea series in 1964.
- DuckDB with httpfs queries a single institution's CSV directly off the Hub, with no download step.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Any pair a publisher does not quote stays absent from the dataset, so inversion and cross-rate code remains in-house, and that code is what an auditor questions.
- decision A team running per-jurisdiction scrapers now has to say what its pipeline adds on top of reading 107 CSVs, because the collecting was most of the job.
- capability Comparing publishers becomes one query: the ECB quotes EUR to USD, the Fed USD to EUR and the Bank of Canada both against CAD, and they differ by a few basis points because they fix at different times.
- exposure Filing evidence sourced this way depends on a third party's collection, and the check available is re-fetching the page named for that institution in sources.json.
Two of the five columns are where the publishers' conventions land: the direction of the pair, and the label in `type`.
The ECB file's row for 2 January 2025 reads `2025-01-02,EUR,AUD,reference,1.6618` [3]. Direction follows the publisher, so if the ECB says 1 EUR = 1.0850 USD, the stored row is `EUR,USD,reference,1.0850` [7]. A filing that needs the other direction inverts it: 1 divided by 1.0850 is 0.92166 at five decimals [2]. Which decimal you stop at, and whether you round the inverse at all, is a rule you write. The maintainers also do not compute crosses, so a bank that never quotes GBP against JPY has no GBP/JPY row [5].
The `type` value is whatever the publisher calls the series [6]. A query filtering `type == "reference"` is therefore a statement about one institution's vocabulary.
The dataset does the fetching for you. "They have to be collected from the source, on the source's schedule, in the source's convention," the maintainers wrote of official fixings [15]. HMRC publishes monthly rates for UK VAT and customs [12]. The ECB reference rate is the default for euro-area statutory reporting, and the Reserve Bank of India reference rate is the benchmark for statutory rupee conversions [13]. The Reserve Bank of Australia, the Bank of Canada and Banco Central do Brasil each publish a daily fixing that is the rate for that jurisdiction regardless of what the interbank market did five minutes later [14].
Across 107 institutions, 14.6 million rows averages about 136,000 rows each [1]. The distribution will not be flat: most institutions publish only a few years of history on their own sites, and where a bank offered more, the maintainers took all of it [11]. The post's pandas example reads one institution's file at a time, on the grounds that 14.6 million rows is a lot of memory for a question about one bank [19]. Polars scanning `rates/*.csv` lazily is the alternative when the question spans several banks [18].
The remaining in-house work is the policy, and the post states it as a procedure: pick the publisher your jurisdiction names, filter to `type == "reference"` or whatever that publisher calls its headline rate, and take the row for the invoice date [20]. If that date is absent, the publisher did not publish that day, and your local rule decides whether you use the previous or the next business day [8].
What to watch
- Whether the maintainers publish an update cadence and a licence for the Hub repo. An auditor relying on it will ask for both first.
- Whether the type labels get documented per publisher, so a filter for the headline rate has documentation behind it instead of a publisher-specific convention nobody wrote down.
- Whether anyone reconciles a filed return against these CSVs and finds a row that diverges from the publisher's own page.