Kubernetes for AI workloads: GPUs, DRA and job scheduling in practice

Why GPUs in Kubernetes are a hot topic right now

According to the CNCF annual survey for 2025, 82% of container users run Kubernetes in production, and 66% of organizations hosting generative models use it to run some or all of their inference. The same survey also shows the other side: 44% of respondents do not yet run AI/ML workloads on Kubernetes, and only 7% of organizations deploy models daily.

In other words, almost anyone can stand up a cluster with a GPU node pool today, but few know how to run AI workloads on it efficiently and repeatably. The reason is largely technical. Kubernetes was designed for CPU and memory, resources that can be divided and rescheduled. A GPU is expensive, indivisible (unless you explicitly make it otherwise), and the workloads running on it have different scheduling needs than a web service.

Over the past year, upstream has moved further in this area than in several years before. What follows summarizes what is genuinely stable as of October 2026, what is beta, and what that means for AKS and EKS.

DRA: what is stable and what is not

The classic device plugin advertises GPUs as an integer: a node has four units of nvidia.com/gpu, a pod asks for one. The scheduler has no idea which card model it is, how much memory it has or how it is interconnected. Everything else is patched in through node labels, node selectors and separate node pools.

DRA replaces this model. The driver publishes devices along with their attributes into ResourceSlice objects, the administrator defines a DeviceClass, and the workload requests a device through a ResourceClaim or ResourceClaimTemplate, selecting it with CEL expressions. The core of this API has been GA since version 1.34 (August 2025); according to the project, further DRA extensions are to land without backward-incompatible changes.

The current version 1.37, according to the official summary of DRA updates, took support for “DRA Extended Resource” to GA. This is crucial for migration: a DRA driver can also serve a pod that requests nvidia.com/gpu the old way, without a device plugin having to run alongside it. Existing manifests and Helm charts therefore do not have to be rewritten all at once.

Here is what a request for a GPU with at least 20 GiB of memory looks like (attribute names are defined by the specific driver; the example matches the NVIDIA driver):

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: gpu-20g
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu.nvidia.com
          selectors:
          - cel:
              expression: device.capacity["gpu.nvidia.com"].memory.isGreaterThan(quantity("20Gi"))
---
apiVersion: v1
kind: Pod
metadata:
  name: inference
spec:
  resourceClaims:
  - name: gpu
    resourceClaimTemplateName: gpu-20g
  containers:
  - name: server
    image: registry.example.com/llm-server:1.0
    resources:
      claims:
      - name: gpu

GPU sharing: time-slicing, MPS, MIG and what DRA does not solve

A whole card for every pod is the simplest and most expensive option. For smaller models, notebooks and development environments, GPUs get shared. The EKS best practices guide for AI/ML describes three mechanisms:

• Time-slicing – workloads take turns on the GPU over time. It requires no physical partitioning, but according to the documentation it “may lack memory isolation”.
• MPS (Multi-Process Service) – multiple CUDA processes run concurrently on one card, with better isolation than time-slicing.
• MIG (Multi-Instance GPU) – physical partitioning of a card into isolated instances with dedicated compute and memory. Only some card generations support it (the documentation names Ampere, Hopper and Blackwell, e.g. the A100 and H100).

DRA can describe and allocate part of a card. What it cannot do is force a container to stay within its allocation. An August post on the CNCF blog sums this up by calling DRA a “promise tracker”: it guarantees that the scheduler does not promise more than the device has, but it does nothing about a container that breaks that promise at runtime. Enforcing a memory limit has to come from another layer – MIG at the hardware level, or a software solution such as HAMi that intercepts CUDA calls.

Job scheduling: Kueue, gang scheduling and the native Workload API

Distributed training needs all its workers at once. When the scheduler places six of eight pods and the remaining two wait for GPUs, six cards sit idle and you pay for them. Gang scheduling prevents this – either the whole group is scheduled, or nothing is.

The second problem is organizational: who may consume how many GPUs, and who waits in the queue. The default scheduler has no answer to that. Kueue solves it as an add-on layer: it decides when a job waits, when it is admitted and when it is preempted. It offers quotas, cohorts that let teams borrow unused capacity from each other, fair sharing and topology-aware scheduling. It integrates with batch Jobs, JobSet, LeaderWorkerSet, RayJob and Kubeflow training jobs, so a single queue works uniformly for both training and multi-node inference.

What is new in 2026 is that part of this logic is moving directly into kube-scheduler. According to the 1.37 summary, the Workload and PodGroup APIs reached beta, as did workload-aware preemption and shared DRA ResourceClaims for PodGroups. A new CompositePodGroup API for hierarchical groups has been added, intended to provide native support for structures such as JobSet and LeaderWorkerSet.

What this means in practice

• Native gang scheduling replaces only part of what Kueue does. Quotas, queues and capacity borrowing between teams remain the job of Kueue or another batch system.
• A beta API may still change before GA. For production training it is safer today to bet on Kueue and try the native APIs in a non-production cluster.
• If you have one team and a handful of cards, a queue may be unnecessary overhead. It starts to make sense once multiple teams or multiple workload types compete for GPUs.

EKS and AKS: where upstream meets reality

Amazon EKS

The EKS documentation recommends the NVIDIA DRA driver for new deployments on Kubernetes 1.34 and later, but with important exceptions. The DRA driver is not supported in EKS Auto Mode, and in Karpenter only with statically provisioned capacity; for dynamic provisioning in Karpenter, the device plugin remains the option. GPU sharing (dynamic MIG, MPS, time-slicing) is marked alpha for the DRA driver and GA for the device plugin.

The documentation also warns that the DRA driver and the device plugin must not run on the same node – the same device may silently be allocated to multiple pods. On the Bottlerocket AMI, the built-in device plugin therefore has to be disabled.

Azure AKS

The AKS engineering blog describes MIG with DRA on a cluster running Kubernetes 1.34 or later. The procedure: create the node pool with the --gpu-driver none option so that the driver stack is managed by the NVIDIA GPU Operator, disable the device plugin in the operator, and install the NVIDIA DRA driver separately. It is a working path, but you manage the whole stack yourself – including version compatibility during cluster upgrades.

Node pools, autoscaling and costs

The biggest item on the bill is usually not a poorly chosen instance type, but GPUs running idle. A few rules that apply regardless of cloud:

• Separate, tainted GPU pools. Regular workloads must not stray onto them; system DaemonSets need tolerations.
• Gentle consolidation. For interruption-sensitive workloads, AWS recommends the WhenEmpty policy in Karpenter with a longer consolidateAfter; the WhenEmptyOrUnderutilized policy may terminate nodes prematurely and prolong job run times.
• Spot only where interruption does not matter. AWS cites discounts of up to 90% versus On-Demand, but explicitly warns that spot is not suitable for stateful workloads that cannot tolerate interruption. Training on spot therefore requires checkpointing.
• Capacity reservations. For P-family instances (and Trainium), AWS offers Capacity Blocks for ML; for the G family, On-Demand Capacity Reservations. For scarce instance types, a reservation is often the only way to get capacity at all.
• Multiple instance types. A pool restricted to a single type will hit regional capacity shortages sooner.
• Scaling to zero. Version 1.37 has HPA scale-to-zero in beta and enabled by default (it requires an Object or External metric); it is worth trying for occasionally used inference endpoints. Expect cold starts, though – pulling the image and loading a large model can take minutes.

Measure utilization, not the number of allocated cards. GPU metrics (utilization, memory in use) belong in the same monitoring as the rest of the platform; without them you cannot tell whether sharing pays off.

AI Conformance as a checklist

The CNCF runs the Kubernetes AI Conformance program, whose requirements are versioned together with Kubernetes. The list for version 1.37 is useful even for companies that do not plan any certification: it describes what an AI platform should be able to do.

The mandatory (MUST) requirements include Gateway API support for managing traffic to inference services, the ability to run at least one gang scheduling solution, cluster autoscaling of accelerator node pools, working HPA for pods with accelerators, accelerator and workload metrics, isolated access to cards, and reliable operation of at least one complex AI operator. The recommended (SHOULD) requirements include GPU sharing, the Gateway API Inference Extension for routing requests to models, and workload sandboxing.

The last point is the one most often underestimated. The GPU stack adds another compatibility matrix to every Kubernetes upgrade; whoever postpones upgrades misses out on the features described above. An overview of supported versions is on the Kubernetes EOL page, and the upgrade process is described in Kubernetes upgrade.

When none of this makes sense

Your own GPU layer in Kubernetes is not mandatory equipment. If a company calls models through a provider's API and hosts nothing itself, it does not need a GPU node pool. The same goes for a handful of experimental jobs a month – a managed cloud ML service will cost less than running operators, drivers and queues.

The opposite extreme is large training clusters, where traditional HPC schedulers still offer more mature tooling for batch work. Kubernetes makes the most sense in the middle: inference of your own or open models, fine-tuning and batch processing in an organization that already runs Kubernetes and wants AI workloads under the same governance, security policies and deployment pipeline as the rest of its applications.

And do not rush into DRA just because it is GA. The device plugin remains supported, and combined with dynamic autoscaling it is often the only supported option today. Migration makes sense once you need something the integer model cannot do: selection by card attributes, finer-grained sharing or device topology.

Key takeaways

• DRA is a stable API, not a finished solution. The core is GA since 1.34, extended resources via DRA since 1.37; driver maturity and autoscaler support vary by cloud.
• DRA allocates, it does not enforce. Memory isolation between workloads on one card comes from MIG or another layer, not from the scheduler.
• On EKS, the provisioning method decides. Auto Mode and dynamic Karpenter mean the device plugin; DRA belongs on static capacity.
• Queues and quotas are Kueue's job. The native Workload and PodGroup APIs are beta in 1.37 and cover only gang scheduling and preemption.
• Idle cards drive costs. Gentle consolidation, spot with checkpointing, capacity reservations and measuring actual utilization have a bigger effect than tuning the instance type.
• The AI Conformance list works as an audit. Even without certification, it shows what a platform for AI workloads should be able to do.