Large-Scale Streaming Operations

Offset lag measures the consumer, not the data: reading freshness from Hudi's own checkpoint

Twilio reported healthy Kafka lag while analytics teams saw lake data hours old. The substitute metric reads the commit file already sitting in S3, and needs nothing from producers or consumers.