Lend and borrow resources across teams with AWS Batch’s quota management for SageMaker Training jobs

With the launch of AWS Batch support for Amazon SageMaker Training jobs, customers gained a new fully managed way to queue, schedule, and retry machine learning (ML) training workloads without building their own infrastructure. Instead, they use the queuing and scheduling features of AWS Batch. As organizations scale their artificial intelligence and machine learning (AI/ML) operations and onboard more teams, coordinating access to shared graphics processing unit (GPU) capacity can introduce additional communication, coordination, and configuration overhead for administrators. This overhead includes setting up separate queues, adjusting capacity limits, and carefully orchestrating the right workloads to get the right resources at the right time. A core value proposition of AWS Batch is helping customers increase utilization of their compute, and as training environments grow, so does the opportunity to do more with existing capacity.

Working backward from customers managing multi-team training environments, we heard a consistent request: give us a way to define allocations per team or project, share idle capacity automatically, and run our highest-priority workloads first. This request led us to build quota management , a new scheduling capability for AWS Batch. Administrators can use it to divide compute capacity into quota shares, organized by team, project, or workload. Each quota share uses configurable resource-sharing strategies that maximize GPU utilization while honoring each team’s allocation when it matters most.

This post shows how quota management works, including how to configure quota shares with lending and borrowing strategies, set up in-share preemption for priority workloads, and get started using the SageMaker Python SDK.

Administrators can allocate and share constrained GPU resources by defining a quota share for separate entities, such as individual teams or projects. Each quota share includes compute quotas (capacity limits) and resource-sharing strategies for idle compute. These strategies determine whether idle compute is reserved until jobs arrive ( RESERVE ), lent to other quota shares that have work available ( LEND ), or both lent when idle and borrowed when the share has work that might benefit ( LEND_AND_BORROW ).

A quota share operates as a virtual queue nested hierarchically within an associated job queue. When scheduling jobs for a job queue, AWS Batch prioritizes quota shares that are under their capacity limits as it iterates through the associated quota shares. If jobs arrive in a quota share that has lent out its capacity, AWS Batch performs cross-share preemption. This reclaims resources for the lending quota share, so administrators don’t need to track down the jobs borrowing capacity and terminate them on their own. If jobs arrive in a quota share that is at or above its capacity limit, AWS Batch checks whether there’s available capacity to borrow from another quota share. If so, workloads can run using otherwise-idle capacity.

You can turn on priority-based in-share preemption when you configure a quota share. Where cross-share preemption reclaims lent capacity as jobs arrive in a lending quota share, in-share preemption preempts lower priority jobs that are in the SCHEDULED, STARTING, or RUNNING state. This allows higher priority jobs at the front of the queue to start running sooner.