Data-Efficient Learning

The 150,000-to-1 gap: scaling is a workaround for a learning mechanism nobody has found

Llama 3.1 was pretrained on 15 trillion tokens. A preteen manages on about 100 million words. Nobody knows how the child does it, and easily available web data may run out in the 2030s.