Machine learning theory

Four substitutions turn a Hopfield update into scaled dot-product attention

Ramsauer and colleagues proved the equality in 2020, three years after attention was published. It holds for a single retrieval step inside a forward pass, and the capacity figures quoted alongside it come from a different model.