Kubernetes JobSet Explained: Coordinating Distributed Workloads

Updated on Oct 12, 2026

10 min read

Distributed training and high-performance computing often need several cooperating groups of Pods to start, discover one another, and finish or recover together. A Kubernetes Job can manage a finite task, but separate Jobs do not by themselves describe the whole multi-role workload. Kubernetes JobSet provides a higher-level API for grouping Jobs under one lifecycle. This guide is for cluster operators and developers running distributed workloads; it covers what JobSet coordinates, where other Kubernetes components still matter, and how to try a small example.

What Is Kubernetes JobSet?

JobSet is a Kubernetes-native API and controller from the Kubernetes SIG Apps community. A JobSet resource describes one or more groups of Kubernetes Jobs and lets the controller manage those child Jobs as a unit. A group can contain Jobs with different templates, such as a coordinator and a worker group, or several groups that use the same template. The JobSet concepts documentation describes this model and the API fields.

JobSet does not replace the built-in Job controller. It uses Jobs as its building blocks: the standard controller creates Pods, retries Pods according to Job configuration, and tracks completions. JobSet adds a parent object for defining groups of those Jobs, coordinating their lifecycle, and supplying networking defaults useful to distributed applications.

The JobSet API is provided by an installed custom resource definition (CRD) and a controller, rather than being built into every Kubernetes cluster. The JobSet project repository publishes the implementation, release manifests, examples, and design proposals. Its objects follow the general resource and status patterns described in the Kubernetes community’s API conventions, but a JobSet is a project-specific API rather than an IETF or other cross-platform workload standard.

The Problem JobSet Solves

One Job is a useful abstraction when all Pods share the same template and can be treated as replicas of one task. Distributed programs often have a more structured shape. A training run might have a coordinator, several workers, and a parameter server. An MPI workload might launch a set of ranks that must know their peers’ addresses. Those roles can require different commands, resources, completion counts, or restart rules.

Without a group-level resource, operators can create multiple Jobs and connect them with scripts, labels, or an external workflow engine. That shifts important lifecycle logic out of Kubernetes objects. A failed worker may need the rest of a training run to stop or restart; a coordinator may need to become ready before workers start; and every Pod may need a predictable name for peer discovery. Separate Jobs do not express all these relationships as one workload.

JobSet makes the group visible to Kubernetes and to other cluster components. It does not make distributed software itself reliable: applications still need suitable initialization, readiness, communication, checkpointing, and retry behavior. The value is a shared orchestration object from which a controller can create and observe the related Jobs.

How JobSet Works

The control path begins when an API client submits a JobSet. The API server validates the resource against the installed CRD and admission configuration. The JobSet controller watches resources, creates child Jobs from their templates, and updates parent status as the Jobs change. The built-in Job controller then manages each Job’s Pods. The scheduler still places those Pods onto nodes, and kubelets run their containers.

Concern	Standalone Kubernetes Job	Kubernetes JobSet
Workload definition	One Job template with its own completions and parallelism	One parent object containing one or more named Job groups
Different roles	Usually requires separate Jobs and external coordination	Separate `replicatedJobs` entries can use different Job templates
Lifecycle view	Status and retries are tracked for that Job	Related Jobs are created and observed through a common parent
Pod discovery	Configure Services or another discovery mechanism	Can create headless Services and stable Pod hostnames for members
Resource placement	Scheduler places each Pod under normal constraints	Same scheduler behavior unless separate queue or gang-scheduling integration is configured

This division of responsibilities matters. JobSet groups and manages Jobs, but it does not replace the scheduler, a workload queue, or the training framework. A single parent object also does not mean all Pods receive an atomic reservation of cluster capacity. Kubernetes Kueue can control queue-based workload admission and quota, while a compatible scheduler integration is needed for any gang-placement guarantees. Kubernetes Dynamic Resource Allocation for GPUs covers a separate layer for requesting accelerator devices.

Components and Key Concepts

JobSet resource: The parent custom resource, usually submitted in a namespace, that gives the workload a name and defines its child Job groups.
ReplicatedJob: An entry in .spec.replicatedJobs with a group name, a replica count, and a standard batch/v1 Job template. Here, replicas means how many Jobs to create for that group; each child Job has its own parallelism and completions settings for Pods.
Controller and child Jobs: The JobSet controller reconciles the parent and owns its generated Jobs. The Job controller remains responsible for the Pods and completion behavior inside each child Job. Labels on children allow operators to query a JobSet’s Jobs and Pods together.
Networking: JobSet can manage headless Services and stable DNS hostnames for Pods in a group. This gives distributed frameworks a discovery mechanism, but applications must still use the correct addresses and ports and handle connection timing.
Startup and success policies: Startup configuration can order replicated groups, such as starting a driver before workers. Success policies can describe which groups count toward completion. These settings express workload lifecycle conditions; they are not a general-purpose workflow DAG.
Failure policies: A failure policy can control what happens when a child Job fails, including restarting the group within configured limits. Restarting containers or Jobs cannot restore application state by itself, so distributed training and HPC programs commonly need checkpoints or another recovery mechanism.
Queues and scheduling: JobSet can be integrated with workload-aware scheduling and queue systems. Those systems handle admission, quota, or placement guarantees that JobSet’s parent-child relationship alone does not provide.

JobSet’s API group and version, such as jobset.x-k8s.io/v1alpha2, must be served by the installed CRD. The exact fields supported depend on that installed release. Operators should use the documentation and examples for the version they deploy rather than assuming a manifest from another release is interchangeable.

Real-World Use Cases

Distributed machine-learning training is a common fit. A JobSet can model different roles with separate templates or run a group of workers from a shared template. The framework still performs gradient synchronization, collective communication, or parameter exchange. Kubernetes provides the process and resource orchestration around that work. For the GPU hardware and framework layer, see the multi-GPU setup guide.

HPC and MPI applications can use a grouped resource to describe ranks, launchers, or other cooperating tasks. Stable Pod hostnames and group-level status reduce the amount of bespoke orchestration an operator has to maintain, while the application remains responsible for its communication protocol and work distribution.

Leader-and-worker jobs can use different replicated-job templates and startup ordering when the workers should wait for a driver or coordinator to become ready. That is useful when the startup sequence is part of the workload contract. If a worker must reserve a GPU or another specialized device, JobSet needs to be paired with the cluster’s device allocation and scheduling setup.

Batch jobs with related phases can use multiple groups when the phases belong to one bounded workload and the JobSet policies express the needed start or completion conditions. For a long-running workflow with arbitrary branching, external signals, timers, or human approval, a dedicated workflow engine may be a better fit. JobSet is an orchestration primitive, not a general workflow language.

Getting Started with JobSet

Before installing JobSet, use a Kubernetes cluster within the supported version range, ensure it has capacity for the controller, and verify that kubectl can access it. The official installation guide documents these prerequisites and the available installation methods. The following pins a release so the installed API and the example are reproducible:

VERSION=v0.12.0
kubectl apply --server-side -f "https://github.com/kubernetes-sigs/jobset/releases/download/${VERSION}/manifests.yaml"
kubectl wait --for=condition=Available deployment/jobset-controller-manager -n jobset-system --timeout=120s
kubectl get pods -n jobset-system

Wait for the controller Pod in jobset-system to become ready before applying a JobSet. The manifest below defines one coordinator Job and one worker Job with two completions. The container commands only sleep to demonstrate object creation and cleanup; a real workload must replace them with application code and its communication settings.

apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: training
spec:
  network:
    enableDNSHostnames: true
  replicatedJobs:
    - name: coordinator
      replicas: 1
      template:
        spec:
          parallelism: 1
          completions: 1
          backoffLimit: 0
          template:
            spec:
              restartPolicy: Never
              containers:
                - name: coordinator
                  image: busybox:1.36.1
                  command: ["sh", "-c", "echo coordinator started; sleep 60"]
    - name: workers
      replicas: 1
      template:
        spec:
          parallelism: 2
          completions: 2
          backoffLimit: 0
          template:
            spec:
              restartPolicy: Never
              containers:
                - name: worker
                  image: busybox:1.36.1
                  command: ["sh", "-c", "echo worker started; sleep 60"]

Save the resource as training-jobset.yaml and apply it after the controller is ready:

kubectl apply -f training-jobset.yaml
kubectl get jobsets
kubectl get jobs,pods -l jobset.sigs.k8s.io/jobset-name=training
kubectl get services -l jobset.sigs.k8s.io/jobset-name=training
kubectl describe jobset training

The label selector narrows inspection to child resources owned by this JobSet. Check events and the status of both the parent and child Jobs if Pods remain pending or fail. A pending Pod can indicate insufficient node capacity, an unsatisfied GPU claim, an image pull problem, or a scheduling policy—not necessarily a JobSet controller failure. When finished, delete the parent resource with kubectl delete -f training-jobset.yaml; confirm that its owned child Jobs and Pods are cleaned up.

The sleep example confirms the custom API and basic controller flow, not that a distributed application can discover peers or recover. Before production use, substitute a real image, configure ports and environment variables, size CPU and memory requests, provision storage if required, and test the application’s failure and checkpoint behavior. Review the release-specific API documentation before enabling startup or failure policies.

Common Misconceptions

“JobSet is built into Kubernetes.” The cluster needs the JobSet CRD and controller installed. A cluster without them will not recognize a jobset.x-k8s.io resource.

“A JobSet guarantees gang scheduling.” Grouping Jobs does not itself guarantee that all required Pods can be placed together before any start. Use the queue or scheduler integration intended for the cluster when workloads require admission, quotas, or gang behavior.

“JobSet implements distributed training.” It creates and manages Kubernetes Jobs. Frameworks such as PyTorch or MPI applications still implement the distributed computation, peer communication, synchronization, and recovery of application state.

Kubernetes Architecture Explained introduces the control plane and scheduler involved in placing JobSet Pods.
Kubernetes Dynamic Resource Allocation for GPUs explains how accelerator claims and device drivers provide hardware to Pods.
Multi-GPU Setup for Machine Learning covers distributed training frameworks and the hardware and software stack they depend on.

Changelog

Initial publication; JobSet API and installation guidance checked against the project documentation and release manifest.