Build1 publisher2 min readPublished
Databricks puts MATCH_RECOGNIZE in its compute with the failure window anchored inside DEFINE
Databricks says the clause is now available in its compute, including Lakehouse Real-Time. In the DEFINE block, FIRST(FAIL.event_time) pins an hour-long window to the first failure of each match.
The Engineer · Build desk

What happened
- Databricks says MATCH_RECOGNIZE is now available in Databricks compute, including Lakehouse Real-Time, and describes it as a regular expression for rows that replaces gaps-and-islands SQL.
- Its security example targets a specific shape: five or more failed login attempts inside a narrow time window, immediately followed by a successful login for the same user.
- Detecting a V-shaped price reversal the old way takes LAG and LEAD comparisons, a running counter that increments on each direction flip to number the islands, and HAVING filters on their boundaries.
- The clause version of that query partitions by symbol, orders by time, and defines the V-trend as a sequence of regex-like states.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- capability Abandonment logic can now be written as a positive assertion that the cart add was the session's last event, so a reviewer reads one pattern instead of mentally simulating a self-join.
- decision Teams maintaining hand-rolled sequence CTEs have to decide whether to port them to a clause whose skip and output-row semantics go undocumented in the post.
- constraint The clause helps only where a partition has a trustworthy ordering column, because every published example orders its rows by event time before defining a state.
- cost Adoption means rewriting SQL that already works. The post gives no timings to price that rewrite against the query it replaces.
FIRST(FAIL.event_time) is the expression to read twice. Databricks puts it inside the DEFINE block, where it names the timestamp of the initial failed attempt. Every later failure is then dynamically checked to fall within 1 hour of that first attempt before the pattern can transition to the SUCCESS state [5]. That reference points back into the match being built. Databricks says COUNT() OVER (PARTITION BY user_id ORDER BY event_time) will tell you how many failures occurred in a time frame. But it cannot easily anchor a sliding time window to the first failure in a specific sequence, or cleanly isolate the sequence once a success occurs [6].
The smallest thing that satisfies the credential-stuffing pattern is therefore six rows, five failures plus the success [11]. Each failure after the first is one comparison against the anchor [5]. Databricks' stated reason for anchoring at all is false positives. Counting attempts alone will not tell you whether the failures sat in a narrow timespan or were spread over a month, or whether a success arrived right after them [15]. The post does not state runtime version requirements or publish timings against the hand-written equivalent [13].
The abandoned-cart case is where I would expect the biggest rewrite. Nothing marks the abandonment; the user simply stops, and traditional SQL proves that negative with NOT EXISTS subqueries, self-joins and window functions [9]. In the pattern, the end-of-partition anchor $ forces the cart add to be the last recorded event in the session, and a time filter supplies the idle window [10].
Whether this lands in your codebase depends on the ordering key. The market-data example partitions by symbol and orders by time before it defines a single state [8], and the premise of the whole post is that SQL otherwise treats rows as unordered sets of facts with no inherent concept of a sequence of events [3]. Where event_time comes off a client clock, the pattern matches what the clock says.
Regular expressions are not famous for readability. The alternative Databricks describes is a query that chains multiple common table expressions together, anchors the time window to the first failure, then checks each subsequent row [14].
Three examples are worked in the post: authorization logs, stock price reversals, and a purchase funnel [12]. All three are sequences over one partition key ordered by event time [8][12]. Databricks describes the clause as letting you describe the sequence you care about directly, like a regular expression for rows. On that description, one SQL clause handles the pattern matching and the gaps-and-islands logic is eliminated [2].
What to watch
- Documentation for AFTER MATCH skip behaviour and per-match output rows, absent from the post, decides whether overlapping login sequences are reported once or repeatedly.
- Whether the clause behaves identically on streaming tables in Lakehouse Real-Time and in batch queries over the same history.
- A published comparison of MATCH_RECOGNIZE against the gaps-and-islands query it replaces, on the same table.