Milvus regex filtering on sealed VARCHAR columns now prunes candidates with FMIndex literals instead of only the longest one
Milvus regex filters on sealed VARCHAR columns now use every bounded literal and its AND/OR links to prune rows via FMIndex, a dev.to post says. RE2 then checks only the surviving candidates, so the saving depends on how selective a pattern's literals are in each segment.
The Engineer · Build desk
Milvus regex filtering. Before, only the longest literal generated candidate rows. Now each AND group uses its lowest-cost literal and all OR branches form combined candidate bitmaps. RE2 rechecks candidates. With no usable literal or an exceeded budget, it scans. No timings are reported.
- precedent Previous approach Extracted only the longest literal substring from a regex pattern to generate candidate rows, claim 1
- decision AND groups The literal candidate with the lowest cost is chosen for each AND group, claim 4
- capability OR branches All OR branches are preserved to form combined candidate bitmaps, claim 4
- capability RE2 matcher Compiled once and stored for reuse as the final recheck filter on candidate rows, claim 5
- constraint Scan fallback Falls back to scanning with no usable literal candidates or an exceeded candidate generation budget, claim 8
- constraint Performance figures The post reports no measured timings or throughput figures for the change, claim 13
| Who | How | Kind | Claim |
|---|---|---|---|
| Previous approach | Extracted only the longest literal substring from a regex pattern to generate candidate rows | precedent | 1 |
| AND groups | The literal candidate with the lowest cost is chosen for each AND group | decision | 4 |
| OR branches | All OR branches are preserved to form combined candidate bitmaps | capability | 4 |
| RE2 matcher | Compiled once and stored for reuse as the final recheck filter on candidate rows | capability | 5 |
| Scan fallback | Falls back to scanning with no usable literal candidates or an exceeded candidate generation budget | constraint | 8 |
| Performance figures | The post reports no measured timings or throughput figures for the change | constraint | 13 |
What happened
- Before the change, Milvus extracted only the longest literal substring from a regex pattern to pick candidate rows.
- Regex filtering on VARCHAR columns scanned full row bytes even when the pattern held selective literals that could have narrowed the candidates.
- For each AND group the planner takes the lowest-cost literal candidate, and every OR branch is kept and merged into the combined candidate bitmap.
- The RE2 matcher is compiled once and stored, then reused as the final recheck on candidate rows.
- The filtering preserves the original input bitmaps and NULL validity.
Why it matters
- capability Alternations such as (foo|bar) can now contribute to pruning, because OR branches are kept and combined into one bitmap instead of being dropped.
- constraint A pattern with no usable literal, or one that exceeds the candidate generation budget, gets no pruning and scans as before.
- decision The described path covers sealed VARCHAR columns, and the planner ranks literals per segment, so the gain has to be measured on a team's own patterns and its own segment mix.
On a sealed VARCHAR column the new path runs in four steps, according to the post [11]:
1. Parse the regex and pull out every bounded literal, keeping the AND/OR structure. 2. Ask the FMIndex for a candidate bitmap per literal. 3. Combine the bitmaps along that structure. 4. Run the compiled RE2 matcher on the surviving rows only.
The post's example pattern is `(foo|bar).*baz`. The old code took `baz` and ignored `foo` and `bar` [10], so two of the pattern's three literals did no pruning [14]. The example is loose, though. All three literals are three characters long [15], so none is the longest, and the post does not say which candidate the new planner keeps for this pattern.
The planner counts distinct literals in each segment to judge selectivity and cost [3]. In our view, that makes the outcome on a pattern like this depend on data. If `baz` is rarer in a segment than the `foo`/`bar` pair, `baz` alone may still be the best filter there. If the pair is rarer, the new path wins.
We think the final recheck is what makes the pruning safe. The bitmaps only have to include every matching row, and RE2 removes the false positives. A FilteredRE2 adapter maps extracted literals to FMIndex candidates using conservative structural checks [9]. Immutable preparations, namely the compiled regex and the literal extraction, are cached thread-locally [6].
The post does not include benchmark figures [13]. For a speedup to transfer to another deployment, the patterns would need bounded literals that are rare in each segment, and candidate generation would need to stay inside its budget [8].
What to watch
- Whether Milvus publishes timings for the change, and on what mix of regex patterns and segment sizes.
- The size of the candidate generation budget that triggers the scan fallback, and whether operators can set it.
- Whether the same candidate-and-recheck routing is extended beyond sealed VARCHAR columns.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence35
- Adoption
- Insufficient
- Hype gap+20
- Incentives
- Insufficient
- Confidence50
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Before the change, Milvus extracted only the longest literal substring from a regex pattern to generate candidate rows for filtering.
- [2]
Regex filtering on VARCHAR columns used to scan full row bytes even when the regex pattern contained selective literals that could narrow candidates.
- [3]
The system counts distinct literals in each data segment to assess their selectivity and cost.
- [4]
For each AND group in the boolean expression, the system chooses the literal candidate with the lowest cost, and all OR branches are preserved to form combined candidate bitmaps.
- [5]
The RE2 regex matcher is compiled once and stored for reuse as the final recheck filter on candidate rows.
- [6]
Immutable preparations such as the compiled regex and literal extraction are cached thread-locally.
- [7]
The filtering preserves the original input bitmaps and NULL validity.
- [8]
When no usable literal candidates are found or the candidate generation budget is exceeded, the system falls back to scanning.
- [9]
A new FilteredRE2 adapter manages pinned RE2 prefilters with conservative structural checks to map literals extracted from regex patterns to FMIndex candidates.
- [10]
For the pattern (foo|bar).*baz, the previous approach would extract just the longest literal, baz, and ignore foo and bar.
- [11]
On a regex filter over a sealed VARCHAR column, the system parses the regex for bounded literals and their boolean relationships, uses FMIndex to generate candidate bitmaps, combines them per the AND/OR structure, and applies the compiled RE2 matcher only to those candidates.
- [12]
The change routes regex filtering on sealed VARCHAR columns through the existing FMIndex candidate-and-recheck path.
- [13]
The post describes the change and its mechanism but reports no measured timings or throughput figures for it.
- [14]
Of the three literals in (foo|bar).*baz, the old approach used one and ignored two.
- [15]
The literals foo, bar and baz in the post's example are each three characters long, so none is longer than the others.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.toAccelerating Regex Filtering in Milvus with FMIndex Candidate Pruning
1 article · October 13, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- Regex query optimizationFollow
- Substring indexingFollow
- Vector Databases and Search EnginesFollow