other

Hopfield Networks is All You Need

2020 paper by Ramsauer and colleagues, presented at ICLR 2021, introducing a continuous modern Hopfield network whose update rule is equivalent to transformer attention and which is proved to have exponential storage capacity.

Current clusters

build1 publisher

Four substitutions turn a Hopfield update into scaled dot-product attention

Ramsauer and colleagues proved the equality in 2020, three years after attention was published. It holds for a single retrieval step inside a forward pass, and the capacity figures quoted alongside it come from a different model.

Publishers:dev.to

Reality

Evidence74
Adoption: Insufficient
Hype gap−8
Incentives62
Confidence66