Tempo 3.1 lets operators with TLS-only Kafka brokers run the 3.0 architecture

Kafka is on the critical path in the 3.0 design. Distributors write to a Kafka topic, block-builders read from it and turn what they read into Parquet blocks in object storage, and live-stores read the same topic and serve recent traces from it ^[4]. A client that cannot authenticate to the broker therefore sits in front of both the write path and recent-trace reads. The 3.0 client supported only SASL PLAIN over an unencrypted connection, so any broker requiring TLS refused it ^[2]. In our view, clusters with such brokers had no way to run the 3.0 design until 3.1 added TLS and four more SASL mechanisms ^[3][14]. Monolithic mode remains Kafka-free, so the client changes do not apply there ^[12].

Redaction is the more careful piece of engineering. The query syntax is deliberately narrow: one spanset filter, equality comparisons on resource.* and span.* attributes, and && or || between them ^[6]. Grafana limits it that way because the rewrite cannot be undone ^[6]. The --start and --end flags confine a run to a time window. That matters, because Tempo halts compaction for the tenant until the redaction has been applied ^[7].

Those flags carry a version condition. A worker on an older version silently disregards the window and deletes all matches in any block it is handed, so Grafana says to use the flags only once every scheduler and worker runs 3.1 ^[8]. A rolling upgrade leaves the fleet mixed for a while. A windowed redaction run in that interval would delete matches outside the window.

The speed claim comes from a narrow test. vParquet5 is the new default block format ^[16]. Grafana says simple metrics queries run close to twice as fast, and the one test the report cites is a warmed-up run of { } | rate() that finished in slightly over half the time ^[17][18]. The span-only fetch path, experimental in 3.0, is on by default for vParquet5 blocks ^[19]. For the number to transfer, your dashboards would need to be mostly queries that simple, your data would need to sit in vParquet5 blocks, and your caches would need to be warm. The report does not say whether blocks written in an older format get rewritten.

Three additions are still experimental: the sampling-aware extrapolate hint ^[11], span pruning on the trace-by-ID v2 endpoint, and the trace diff endpoint ^[13]. We would take the Kafka client options, redaction and the new block format with the 3.1 upgrade and wait on those three.