Apple's SimpleDesign drops the tokenizer stage from protein co-design

Somebody on the team owns the tokenizer. Under the usual co-design recipe, that person trains an autoencoder to turn protein structures into discrete tokens, keeps the checkpoint, and reruns it before the generative model sitting on top can be retrained ^[4]. Apple's paper skips that step and produces amino acid sequences and continuous 3D coordinates from a single end-to-end training process ^[3].

"We hypothesize that this multi-stage training is not necessary to obtain performant co-design models and thus present SimpleDesign, an effective multi-modal protein design model trained directly in the data space," the researchers wrote ^[5].

The training set was more than 2 million protein sequence-and-structure pairs, taken primarily from the AFESM dataset, which combines predicted structures from the AlphaFold Database with additional samples ^[6]. Both halves of every pair were corrupted during training: amino acids hidden behind mask tokens, noise added to the coordinates, with the extent of each varied ^[7]. A mostly intact sequence beside a heavily corrupted structure gives the model a folding problem, and the reverse gives it inverse folding ^[8]. Corrupt both partway and it is training for co-design ^[9].

The change a team can count is retraining runs. The two-stage recipe needs the autoencoder finished before the generator starts ^[4], so a fresh data cut costs two runs; the SimpleDesign recipe costs one ^[16]. The single-stage run also carries no token vocabulary, so there is nothing to keep aligned with the data ^[3].

Apple reports the simpler pipeline held up: competitive results across protein co-design, structure generation and sequence generation benchmarks ^[10], plausible generated structures, and sequences generally as good as or better than those from most competing multimodal models ^[11]. The 9to5Mac account puts those outcomes in words and does not include figures or name the models compared ^[17].

The tidy version of this line of work is that plain Transformers plus flow matching can hold their own against heavier specialist stacks. That version is documented for SimpleFold, published last September, which used a flow-matching model with general-purpose Transformer blocks and avoided some of the more computationally expensive techniques used by protein folding models such as DeepMind's AlphaFold ^[13][14]. For SimpleDesign, the comparison the researchers draw is with multi-stage latent-space training ^[5], and 9to5Mac attributes flow matching to the folding paper ^[18]. Diffusion is what that account contrasts flow matching with as a technique ^[15].

So this is not reading for a group choosing a production protein design model this quarter ^[17]. It is for teams running a tokenizer in front of a generator for any paired-modality data, and the useful question is what the discrete tokens do for them. If the tokens exist because the generator was built to consume tokens, that is overhead a single-stage run removes ^[16]. If downstream tooling indexes, searches or edits those tokens, the extra stage is the price of that tooling, and Apple's paper makes its claim about training ^[5].