The EU AI Act pushes training-data provenance down into the crawler

A robots.txt file tells you what a site allowed at the moment you fetched it. Fetch it in March and you know what the site allowed in March. Re-fetch it in October to answer a question about a March crawl and you learn what the site allows in October. The only record that covers the fetch you actually made is the one the crawler wrote at the time: the URL, the fetch time, the directive it read, and the decision it took. Recording where each record or source came from, and when, is the cheapest write in the whole crawl ^[13]. The dev.to post's checklist says the same thing in operational terms, telling teams to make robots.txt and machine-readable TDM reservations "a real gate in collection, not something checked manually and inconsistently" and to log the decision so it can be demonstrated afterwards ^[7].

Provenance runs the same direction. Once a dataset is a merged pile of text, the origin of a given passage is not recoverable by inspection. "You cannot document a training set after the fact if you never captured its origins," the post says ^[8].

The published number is a ceiling with two limbs: up to 15 million euros or 3% of global annual turnover, whichever is higher ^[2]. Divide 15 million by 0.03 and the limbs meet at 500 million euros of turnover ^[11]. Under that, the flat cap is the binding one; over it, the percentage is. The separate tier for prohibited practices, 35 million euros or 7%, crosses at the same 500 million ^[3][12].

GDPR is not displaced by any of this, according to the post ^[15]. It cites European data-protection guidance to the effect that scraping publicly accessible data does not exempt you from GDPR when the data includes personal information, so a page being public does not make the personal data on it free to train on ^[9]. The guidance is not named ^[17]. The engineering item that falls out of it is the ability to identify and handle personal data in what you collected ^[14].

For the transparency obligation, the artifact is public: providers have to publish a sufficiently detailed summary of training content on a European Commission template that the post says is now finalized and mandatory ^[5]. If the provenance is there per record, producing that summary is a query. If it is not, someone writes it from memory.

The record here is a single dev.to post, and it states that it is not legal advice ^[10]. It also says most of the Act is not about data collection, and that three of its requirements turn into engineering work when training data comes from the web ^[16]. Its checklist has four items: capture provenance at the point of collection, gate the crawler on opt-out signals and log the decision, filter or flag personal data, and be able to produce the training-data summary ^[18].