Skip to content

Topic

Real-time inference latency

The delay between a user's input and a model's response, which in interactive systems decides whether an exchange feels like conversation.

Current clusters