AI21 Nov 20262 min read

AWS Launches SageMaker HyperPod Gateway to Route AI Workloads Across Regions

Amazon SageMaker HyperPod has added a global gateway that spreads AI inference traffic across clusters and regions to keep expensive GPUs busy and handle failovers automatically.

Listen to this storyAbout 3 min, read by your device
Picture made with AI

Amazon Web Services has introduced the SageMaker HyperPod Global InferenceInferenceThe process of running live data through a trained artificial intelligence model to make predictions or generate outputs. Open in glossary Gateway. The managed routing service distributes artificial intelligence inference workloads across multiple clusters, whether located in a single AWS region or spread across different parts of the world. The tool aims to cut idle computing time and prevent traffic bottlenecks for large AI models.

Solving Multi-Cluster Traffic Imbalances

As artificial intelligence models grow, finding enough graphics processing units in a single data centerData CenterA large facility housing computer systems and associated components for storing and processing massive amounts of data. Open in glossary becomes more difficult. Teams often spread their computing capacity across multiple clusters and regions. Without automated routing, client applications must track individual endpoints for every model and cluster. This setup often causes traffic jams in one region while hardware in another sits unused.

The Global Inference Gateway provides a single access point. Instead of forcing developers to build their own domain name systems and load-balancing networks, the service handles routing over the private AWS backbone. If a cluster goes down, the gateway moves traffic to a healthy one on its own without requiring client changes.

How Routing Works

Setting up the service involves publishing clusters through their configuration files and grouping them under one gateway endpointEndpointA web address where software applications send and receive data requests. Open in glossary. Inside your account, the setup creates a network load balancer and a virtual private cloud endpoint service. The gateway then authenticates requests, checks system health, and forwards incoming traffic through private links.

To decide where to send each request, the gateway scores available clusters using four signals:

  • Region proximity to reduce latencyLatencyThe delay between a request being sent and a response being received. Open in glossary
  • Queue depth to avoid busy clusters
  • Available key-value cache and GPUGPUA chip that does many small sums at once. First made for graphics, now the main chip for AI. Open in glossary memory headroom
  • Recent error rates

Users can customize the weight of each factor, using values from zero to one that must add up to one. By default, regional proximity carries the highest weight, followed by cluster queue depth.

Two-Tier Traffic Management

The global gateway acts as a top-level routing layer. Once a request arrives at a chosen cluster, the existing local inference gateway takes over. That internal gateway places the task on the best individual model pod based on local queue depths and memory use. This multi-tier setup ensures both cross-region balance and pod-level optimization.

How does this story land?

One tap per reader.

Quick poll

Is cross-region routing worth the extra network latency for AI inference?

One tap. No sign-up.

Ask this story

Still have a question?

Answers use only what is in this story. They are written by AI and can be wrong.

Sources

  1. Artificial Intelligence, Introducing Amazon SageMaker HyperPod Global Inference Gateway: One endpoint across Regions and clusters
Companies in this storyAmazonBagelcodeWords explainedInferenceData CenterEndpointLatencyGPU
Up next in AIAWS adds managed extensions to SageMaker HyperPod to fix Slurm drift2 min read

Keep reading

All AI stories