Databricks' Funke parses HL7v2 messages into native Spark columns without converting to FHIR first

Funke's map keys every value by position ^[4]. That matches how HL7v2 is built, with segments, fields, components and subcomponents separated by characters the message declares in its own header ^[1]. It also means the parser records where a sender put a value. It has no view of what the sender meant. Databricks' own post says the standard is loose enough that messages in production differ from one sender to another ^[2]. If two lab systems carry the same unit in different components, Funke keeps both faithfully, at two different addresses. The pipeline defines two tables and leaves the gold tables to the user ^[9].

The name is German for spark, a nod to its predecessor Smolder ^[15]; at least the fire is heading the right way. The parsing itself is careful work. It takes the field, component, repetition and subcomponent separators from each message's MSH header and applies the standard escape sequences ^[5]. Databricks describes the parse as designed to be lossless, with the goal of keeping the original structure as far into the pipeline as possible ^[7]. The post does not say what the parser does with a message that breaks the encoding rules.

Bronze keeps each decoded message next to an MD5 hash, an insert timestamp and a message ID ^[6]. Silver adds a single hl7 column holding the parse ^[10]. Keeping the raw text in bronze is the design choice I like most. If the parser mishandles one sender, I'd expect the fix to be replayable against bronze history without asking anyone to resend a feed.

Databricks frames Funke against two workarounds. Converting to FHIR first adds a translation layer and can drop detail that has no clean FHIR equivalent, the post says ^[11]. A third-party flattening engine means paying another vendor and moving data off the platform, and its wide tables lose the granular structure underneath ^[12]. Funke's answer is to postpone the shape decision. The post says users decide which fields matter for each use case instead of accepting the shape a converter or vendor chose ^[14]. With Funke, that shape is gold-table code the team's own engineers write and maintain ^[9].

Databricks says a team can go from nothing to a scalable streaming HL7 pipeline in minutes ^[18]. That figure covers standing up bronze and silver. The per-sender mapping into gold is separate work.

For Funke to replace a flattening vendor in a given shop, two things have to be true. Its parser has to accept every sender's real traffic, including messages that bend the spec. The team also has to own the per-sender mapping that previously lived in the vendor's tables. The first condition can be tested before any migration: replay a month of each feed into silver and count the messages that fail to parse or put a field where the gold code does not expect it.