Loading today’s stories
Build1 distinct publisher2 min readUpdated
Meta's new training chip treats collectives as the scarce resource and moves the network into the package. The design only pays off if embeddings, not FLOPs, set the pace.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
The over-provisioning is the tell. The near-memory compute blocks sit at the chip edges next to HBM and cache, each reducing at 128 bytes per cycle, and Meta puts them together at more than 2.8 TB/s of reduction throughput [9][c8b]. That is roughly 2.3 times the chip's entire network bandwidth [1]. Build it that way and an AllReduce can never stall on arithmetic; the wire becomes the only limit, and bandwidth is the thing you can go and buy more of. Each message engine also carries its own RISC-V core to sequence work and its own route to a NIC [c8b], so the reduction path never asks the processing element grid for a slot. On a GPU the same work arrives as NCCL kernels, and those kernels occupy the streaming multiprocessors that the training step also wants [7].
The parameter distribution is what makes this worth silicon. When embedding tables hold over 99 percent of a model's parameters [3], hybrid parallelism issues AllReduce, AllToAll and AllGather traffic constantly across hundreds of accelerators [c3b]: many small operations rather than a few enormous ones. That is why 800 nanoseconds is a number worth chasing. MTIA 300's express doorbells make the work request write serve as its own doorbell, removing a memory read and saving about that much per operation [6]. HCCL compiles each collective into subgraphs of work-queue entries with explicit dependencies instead of steering it from the host at runtime [c11b], so per-operation overhead stops tracking host involvement.
The asymmetry to watch is inside the rack. The same NICs deliver up to 1 TB/s of scale-up bandwidth across 16 nodes and 200 GB/s of scale-out across racks [5], a factor of five [2]. A job whose chatty collectives fit in one rack gets the best case; one that spills across racks meets a fifth of the bandwidth with the same offload engines sitting behind it. The 1.2 TB/s headline, incidentally, is exactly twelve 800 Gbps NICs converted at eight bits to the byte [4], so there is no hidden margin in that figure.
One caution across all of it. These are Meta's own measurements published on Meta's own engineering blog, the comparison is to unnamed general-purpose GPUs [10], and the silicon detail is deferred to an ISCA 26 paper [12]. Nobody outside Meta has run the same ranking job on both sides of the comparison.
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
Meta states that unlike large language models, which need enormous floating-point throughput, recommendation models are bottlenecked by the need for fast and efficient communication between the accelerators that train them.
HCCL, Meta's communication library, was co-designed from scratch with MTIA 300 rather than adapted to it, making communication what Meta calls a first-class citizen in the chip's design.
Rather than driving communication from the host during execution, HCCL compiles each collective into a complete set of subgraphs: arrays of work-queue entries with explicit dependencies.
MTIA 300 is the first of Meta's family of in-house training and inference accelerators optimized for training ranking and recommendation models.
Embedding tables in these recommendation models can contain over 99% of the model's parameters.
The resulting hybrid parallelism generates frequent AllReduce, AllToAll and AllGather collectives across hundreds of accelerators.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Detailed but wholly first-party
The architectural disclosure is unusually specific and internally consistent: NIC count and link rate reconcile exactly to the claimed 1.2 TB/s, and the 2.8 TB/s reduction figure matches the stated 'more than double I/O'. That specificity earns credit. But every number originates with the designer, the GPU comparisons name no baseline configuration or methodology, the referenced ISCA 26 paper is outside the cluster, and no independent party has measured the chip. Evidence quality is therefore capped well below the level of detail on offer.
In production inside Meta, nowhere else
Adoption is real but narrow and unquantified. Meta discloses one concrete production workload — a 150-billion-parameter recommendation model on 40 accelerators — plus an in-rack bandwidth figure, which is more than a lab demo but far short of fleet-scale evidence. No unit volumes, rack counts, rollout timeline, or share of recommendation training on MTIA 300 are given, and because the chip and HCCL are internal there is no external adoption surface at all.
Design detail solid, comparisons oversold
The gap is moderate and concentrated in the comparative framing rather than the engineering. Claims like 'near-perfect isolation', 0.5% versus 20%+ degradation, and 3.9x faster communication are stated without a named GPU baseline, library version, or methodology, and they carry the whole strategic argument; the fleet-sizing implication that follows from them inherits the same weakness. Against that, the architectural disclosures are specific, checkable and consistent, and Meta is explicit that the design targets recommendation workloads rather than LLMs — restraint that keeps the gap from being large.
Designer, benchmarker and publisher are one party
Meta authored the chip, ran the comparisons, and published the only account of them on its own engineering blog. The post's purpose is to argue that in-house silicon beats general-purpose GPUs on Meta's most commercially important workload — a claim that supports capex justification, silicon-team recruiting, and negotiating leverage with merchant accelerator vendors. Nothing here is falsifiable by a reader, and no disconfirming detail (cost, power, workloads where the design loses) is offered.
Confident on architecture, not on advantage
Confidence is moderate and asymmetric. That MTIA 300 exists, is in production on at least one large recommendation model, and is built the way described can be held with reasonable confidence — the disclosure is detailed, dated, internally consistent, and comes from the party that would know. The size of the advantage over GPUs, its durability, and its economics cannot be held with confidence at all, given one first-party source, no named baseline, and no cost or fleet data.
invest
Meta's four MTIA generations show inference leaking from Nvidia one workload at a time1 distinct publisher
build
The generative recommender's real constraint is not the model, it is the memory1 distinct publisher
build
Dual 3090s, no NVLink: the serving stack broke long before the model did1 distinct publisher
invest
Broadcom's $100B Chip Raise Turns AI Capacity Into a Credit Question1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.