The generative recommender's real constraint is not the model, it is the memory

NVIDIA has published a walkthrough of the shift from embedding-similarity recommenders to generative ones, framing recommendation as predicting the next action or item in a large catalog given a sequence of user histories ^[2]. The post covers the architectural change, the production problems it creates, and points at two pieces of NVIDIA tooling, recsys-examples and nv-embedding-cache, as the answer ^[3]. The interesting part is not the objective function. It is that the post's own account of the constraints puts the bottleneck in the memory hierarchy, not the model.

The architectural argument is straightforward. Generative recommenders treat the problem as sequence modelling in the manner of an LLM rather than geometric similarity between user and item vectors ^[12]. NVIDIA's stated upside is that more homogeneous, transformer-shaped architectures track scaling laws better, could collapse retrieval and ranking into one model, and sit closer to the LLM tooling ecosystem ^[13]. Two approaches implement the objective: Hierarchical Sequential Transduction Units and Semantic IDs ^[14].

HSTU came out of Meta in 2024 ^[15]. It represents input as a per-user sequence of interleaved items and actions ordered by timestamp ^[16], and drops explicit feature engineering in favour of learned sequential representations from attention over user-item interactions ^[17]. The attention block is not standard: softmax normalisation is replaced with SiLU-based weighting, relative attention bias is added, and elementwise gating is applied before the output projection ^[18]. According to NVIDIA, those changes preserve magnitude information across long sequences while allowing more kernel fusion and lower-latency inference ^[19]. The framing makes user histories analogous to next-token prediction, with learning signal available both within a sequence and across batches ^[21].

Now the operational reality. User histories are the primary data type, a mix of categorical and continuous features that change frequently ^[4], and at industry scale they reach the order of terabytes or petabytes every day ^[5]. NVIDIA states plainly that even on the highest-end accelerators, data of that size will not fit GPU high-bandwidth memory, which introduces bottlenecks in both training and inference ^[6]. Meanwhile the serving budget is unforgiving: these models run online to millions of users under strict SLAs where small latency increases hurt ^[10], and unlike LLM workloads that can absorb autoregressive decoding latency, a recommender has to retrieve and rank thousands of candidates in a few milliseconds ^[11]. Put those two together and the engineering job is staging a working set that lives outside HBM and fetching the right slice of it inside a millisecond-scale budget ^[22]. That is a caching and data-movement problem wearing a model architecture costume.

The data problems the post lists are the ones that make caching hard rather than optional. Interaction distributions are heavily skewed to popular items, so most of a catalog gets little training signal ^[7][8], and new users or items arrive with no history at all, forcing embeddings to be inferred from a thin feature set with a real risk of poor early recommendations ^[9]. Sparse next-item prediction over a large corpus also brings full-softmax cost and weak long-tail signal ^[20].

What to watch: whether nv-embedding-cache ships with measured hit rates, tail latency under an SLA, and host-to-device bandwidth numbers, because the post as written names the tools without quantifying them ^[3]. Also watch whether HSTU's fusion claims hold up as throughput on real catalogs rather than as kernel-level assertions ^[19].