Cache-aware routing can cut inference latency and cost for production LLM endpoints.
Amazon SageMaker AI real-time endpoints now support KV cache-aware routing, sending each request to the instance holding the longest cached match for its prompt, as an alternative to prefix-aware routing.
Read it at AWS Machine Learning →