Introducing Amazon SageMaker HyperPod Inference Gateway
- AWS: 110 events in the last 90 days
- Previous: earlier the same day · New low-cost burstable Amazon EC2 T8i instances are generally available
What happened
Amazon SageMaker HyperPod Inference Gateway is a Kubernetes-native, GPU-aware routing add-on for Amazon EKS. It uses real-time GPU signals to send each inference request to the best-suited pod, cutting first-token latency by up to 82% with no changes to your model servers or client applications.
Summary assembled by rule from the sources below
Why it's spreading
Sources
Official
- Introducing Amazon SageMaker HyperPod Inference GatewayAWS Machine Learning