Build1 publisher2 min readPublished
An unindexed join field turns one million customers into one million collection scans
MongoDB caps each aggregation stage at 100MB of RAM. How many documents reach the first blocking stage decides whether a pipeline streams or spills, and that count depends on where $match sits and whether the join field is indexed.
The Engineer · Build desk

What happened
- MongoDB limits every aggregation stage to 100MB of RAM by default, and a stage that exceeds the limit fails the whole operation unless the query is run with allowDiskUse set to true.
- A $match on indexed fields at the head of a pipeline is merged with the initial FIND and executed during the collection scan, so fewer documents are loaded into the aggregation engine.
- $sort and $group buffer their input and block until all of it is consumed, unlike streaming stages that pass documents along one at a time.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint The 100MB cap sits on the stage, so tuning a blocking $group or $sort means changing what arrives at it: fewer documents, fewer fields. Everything that helps happens upstream of it.
- decision allowDiskUse picks which failure the operator ships: an error the client sees, or a query that completes after serialization and temporary file I/O.
- exposure The post attributes collection-level locks and cluster-wide degradation to badly structured pipelines, so one reporting query can reach every other workload sharing the cluster.
- capability Indexing the foreign join field replaces a per-document collection scan with a lookup, and that fix belongs to whoever owns the orders collection.
Put a $match on an indexed field first and the query planner folds it into the initial FIND, so the filter runs as part of the collection scan and fewer documents ever enter the aggregation engine [2]. Drop the same $match below a $lookup or a $group and it cannot be pushed down at all; it filters whatever the previous stage left in memory [3]. The predicate and the index are identical in both pipelines.
The worked example in the dev.to post is a customers-to-orders join with no index on orders.customerId, and MongoDB falls back to a full scan of the foreign collection for every document in the source stream [4]. With 1 million customers and 10 million orders, the post counts 1 million collection scans [5]. Taken literally, each scan is a pass over 10 million documents, so the pipeline reads 10 trillion documents [1].
Reordering does not touch that number. The scan count is set by the missing index on the foreign field [4].
Fan-out compounds. The post's other example is a customer with 10 orders of 5 line items each, which becomes 50 documents after $unwind [6]. If the average customer in that million-document collection looked like that, the $group downstream would receive 50 million documents [2]. Against a 100MB stage budget that is about 2 bytes per document [3], and the post's rule is that a $group on a non-indexed field processes every one of them in memory [8]. Hence the advice to drop fields with $project or $addFields before the expensive stages [13].
The post's author calls allowDiskUse "a safety net, not an optimization strategy" and says a pipeline that needs to spill is almost always an opportunity for restructuring [10]. Spilling means serialization, file I/O and temporary file management, which the post puts at orders of magnitude slower than in-memory processing [11]. It is also one line of code and no schema change, so it survives code review.
The last stage of the typical pipeline the post prints is a $sort on total descending, where total was computed by the preceding $group [15]. No index covers a field that did not exist until the previous stage, so by the post's own rule that sort loads all documents into memory and runs O(n log n) [4]. Two things have to be true for the ordering advice to transfer to your cluster: the leading $match has to be index-backed, and the sort keys have to be stored fields. A pipeline that sorts on a computed total blocks wherever you put it, and the only variable left is how many documents arrive [7].
What to watch
- Measured explain output or timings for the unindexed $lookup example, which the post does not publish.
- Whether $group memory tracks input document count or distinct group keys, which decides how close the 100MB cap really is.
- Any change to the 100MB per-stage default or spill behaviour in a MongoDB release, which would reset every ordering rule here.