{"id":15773,"date":"2026-11-12T16:33:37","date_gmt":"2026-11-12T08:33:37","guid":{"rendered":"https:\/\/ai-stack.ai\/ai-training-at-scale-slurm-on-k8s"},"modified":"2026-11-12T17:41:08","modified_gmt":"2026-11-12T09:41:08","slug":"ai-training-at-scale-slurm-on-k8s","status":"publish","type":"post","link":"https:\/\/ai-stack.ai\/en\/ai-training-at-scale-slurm-on-k8s","title":{"rendered":"Scaling AI Training: How AI-Stack Slurm on K8s Tackles 3 Key Challenges"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">As large language models continue to grow, AI training is shifting from single servers to large-scale, multi-node distributed architectures. Scale brings more compute, but it also brings three challenges: resource scheduling, operations management, and fault resilience. AI-Stack Slurm on Kubernetes deploys Slurm, the leading HPC job scheduler, on the Kubernetes container platform. Slurm handles job scheduling while Kubernetes handles containerized operations, making it a key architecture for enterprises tackling these challenges.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Trend: AI Training Is Moving to Multi-Node, Large-Scale Clusters<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Large language models keep growing in both parameter count and training data. Meta&#8217;s Llama 3 405B, for example, was <a href=\"https:\/\/arxiv.org\/html\/2407.21783\" target=\"_blank\" rel=\"noopener\">trained on up to 16,000 H100 GPUs at once<\/a>. When a model or dataset exceeds what a single server can handle, GPUs across multiple servers must work together on the same training job. This is multi-node distributed training. Because nodes exchange data constantly and depend closely on one another&#8217;s progress, scaling from one machine to many adds not just GPUs but coordination complexity.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">At the platform layer, Slurm and Kubernetes are the two dominant options. <a href=\"https:\/\/slurm.schedmd.com\/quickstart.html\" target=\"_blank\" rel=\"noopener\">Slurm<\/a> is an open-source cluster management and job scheduling system built for large batch workloads. <a href=\"https:\/\/blogs.nvidia.com\/blog\/nvidia-acquires-schedmd\" target=\"_blank\" rel=\"noopener\">According to NVIDIA<\/a>, more than half of the top 10 and top 100 systems on the TOP500 list of supercomputers run Slurm. Kubernetes, meanwhile, is the mainstream platform for managing containerized services in the enterprise, and its community continues to strengthen support for distributed workloads: native <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/scheduling-eviction\/gang-scheduling\/\" target=\"_blank\" rel=\"noopener\">gang scheduling<\/a> reached beta in v1.37.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Yet mature platforms do not mean scaling is frictionless. Once training expands to dozens or hundreds of GPUs, enterprises face three major challenges.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Challenges: Three Obstacles to Scaling AI Training<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Challenge 1: Resource Scheduling \u2014 Distributed Jobs Can Deadlock<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">A training job needs multiple nodes to be ready at the same time, but the Kubernetes <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/scheduling-eviction\/kube-scheduler\/\" target=\"_blank\" rel=\"noopener\">default scheduler<\/a> places Pods one at a time instead of scheduling all of a job&#8217;s Pods together. Suppose a cluster has 4 GPUs and Jobs A and B each need 4. If each job gets 2, neither can start, and neither releases the GPUs it holds (see Figure 1).<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img data-recalc-dims=\"1\" fetchpriority=\"high\" decoding=\"async\" width=\"1024\" height=\"417\" data-attachment-id=\"15774\" data-permalink=\"https:\/\/ai-stack.ai\/en\/ai-training-at-scale-slurm-on-k8s\/slurm-on-k8s-gang-scheduling-deadlock-en\" data-orig-file=\"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/11\/8c0e83ab.png?fit=1600%2C652&amp;quality=100&amp;ct=202603031250&amp;ssl=1\" data-orig-size=\"1600,652\" data-comments-opened=\"0\" data-image-title=\"slurm-on-k8s-gang-scheduling-deadlock-en\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/11\/8c0e83ab.png?fit=1024%2C417&amp;quality=100&amp;ct=202603031250&amp;ssl=1\" src=\"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/11\/8c0e83ab.png?resize=1024%2C417&#038;quality=100&#038;ct=202603031250&#038;ssl=1\" alt=\"Comparison of a distributed training deadlock caused by Kubernetes' default pod-by-pod scheduling and gang scheduling, which avoids it\" class=\"wp-image-15774\" style=\"aspect-ratio:2.459016393442623\" title=\"slurm-on-k8s-gang-scheduling-deadlock-en.png\"\/><figcaption class=\"wp-element-caption\">Figure 1. Pod-by-pod scheduling in Kubernetes can cause deadlock; gang scheduling prevents it<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The solution is gang scheduling: a job starts only after all of its Pods have secured resources. Native gang scheduling in Kubernetes has reached beta but is not enabled by default, so check the version and configuration when evaluating a platform.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Challenge 2: Operations Management \u2014 Traditional Slurm Clusters Depend on Manual Work<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Slurm excels at scheduling, but traditional Slurm clusters typically run directly on bare metal, which creates three operational pain points:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Manually maintained environments<\/strong>: Drivers, libraries, and software versions must be managed by hand and frequently conflict.<\/li>\n\n\n\n<li><strong>No self-healing<\/strong>: When a component or node fails, operators usually have to step in.<\/li>\n\n\n\n<li><strong>No flexible resource sharing<\/strong>: Training clusters are hard to share with inference services and development environments.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Switching to Kubernetes solves environment consistency and self-healing, but existing Slurm commands and job scripts don&#8217;t carry over directly. For organizations with established Slurm users, the switching cost remains.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Challenge 3: Fault Resilience \u2014 More Nodes, More Interruptions<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/arxiv.org\/html\/2407.21783\" target=\"_blank\" rel=\"noopener\">Meta&#8217;s Llama 3 technical report reveals<\/a> that during 54 days of training Llama 3 405B, there were 419 unexpected interruptions, roughly one every three hours. GPU issues accounted for 58.7% of them, making them the leading cause. Enterprise clusters are far smaller, but the principle holds: if each node has just a 0.1% chance of failing per day, the probability that 100 machines running for 30 days see at least one failure approaches 95% (an illustrative assumption that treats node failures as independent). <a href=\"https:\/\/arxiv.org\/abs\/2410.21680\" target=\"_blank\" rel=\"noopener\">An analysis of two large machine learning clusters by Meta researchers<\/a> also shows that large jobs are hit significantly harder by failures.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Why Aren&#8217;t Checkpointing and Kubernetes Self-Healing Enough?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The most common safeguard is checkpointing: periodically saving training progress, model parameters, and optimizer state so a job can resume from the latest checkpoint after a failure. Checkpoint frequency is a trade-off. Checkpointing too often eats into training time, while checkpointing too rarely means losing more progress with each failure. The optimal interval depends on write cost and failure rate: the more nodes and the more frequent the failures, the shorter the interval should be.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">However, checkpointing only keeps progress from being lost entirely; it does not shorten recovery. After a failure, the whole job must still be requeued, reallocated, and reinitialized, which often takes hours. Kubernetes self-healing cannot close this gap either, because recovering a training job requires three things: all nodes must reconnect and load the same checkpoint, a replacement node must be available immediately, and the faulty hardware must be isolated first so work doesn&#8217;t land on the same node again. All of this calls for a recovery mechanism designed specifically for training.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">INFINITIX AI-Stack: An Enterprise-Grade Slurm on K8s Solution<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/ai-stack.ai\/en\">AI-Stack<\/a> is an AI infrastructure management platform built on Kubernetes container technology, and Slurm on K8s is part of its multi-node training capabilities. AI-Stack addresses each of the three challenges above:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Resource scheduling<\/strong>: Training jobs are scheduled by Slurm, while other distributed workloads on the platform use gang scheduling by default to prevent deadlocks.<\/li>\n\n\n\n<li><strong>Operations management<\/strong>: Slurm components are deployed automatically as containers, with a graphical management interface and comprehensive monitoring.<\/li>\n\n\n\n<li><strong>Fault resilience<\/strong>: Checkpoint storage mounting is supported, and Automatic Training Recovery replaces failed nodes with warm standby nodes.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">For researchers, daily workflows barely change: they still connect via SSH and submit jobs. For operators, Slurm components run as containers and can be deployed and maintained the Kubernetes way. For managers, training no longer needs a dedicated cluster and can share the same underlying resource pool with inference and development. Slurm on K8s solves resource scheduling and operations at the architecture level, while fault resilience is handled by the platform&#8217;s Automatic Training Recovery.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th><strong>Category<\/strong><\/th><th><strong>Traditional Slurm cluster<\/strong><\/th><th><strong>Kubernetes alone<\/strong><\/th><th><strong>Slurm on K8s<\/strong><\/th><\/tr><\/thead><tbody><tr><td><strong>Job scheduling<\/strong><\/td><td>Built for large-scale batch jobs<\/td><td>Strong for online services; batch scheduling needs add-ons<\/td><td>Slurm schedules training jobs<\/td><\/tr><tr><td><strong>MPI\/HPC workflows<\/strong><\/td><td>Native support<\/td><td>Requires adaptation<\/td><td>Existing job submission workflow preserved<\/td><\/tr><tr><td><strong>Environment consistency<\/strong><\/td><td>Manually maintained; version conflicts common<\/td><td>Containerized, consistent, and portable<\/td><td>Slurm components deployed as containers<\/td><\/tr><tr><td><strong>Operations<\/strong><\/td><td>Relies on manual intervention<\/td><td>Self-healing, automated operations<\/td><td>Kubernetes deploys and maintains components<\/td><\/tr><tr><td><strong>Resource sharing<\/strong><\/td><td>Usually dedicated hardware<\/td><td>Unified abstraction for network and storage<\/td><td>Training, inference, and development share one pool<\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\">Table 1. Comparison of traditional Slurm, Kubernetes alone, and Slurm on K8s<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Built-In Gang Scheduling Prevents Deadlocks<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Beyond Slurm clusters, other distributed workloads on the platform are protected too. AI-Stack&#8217;s built-in batch scheduler uses gang scheduling by default, starting a job&#8217;s Pods together only when all of them are ready, which prevents deadlocks. A consolidated placement policy can also be enabled to reduce resource fragmentation. As a result, training, inference, and development workloads are each scheduled correctly within the same resource pool, delivering on the resource-sharing goal of Slurm on K8s.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Automatic Training Recovery: Warm Standby Nodes Restore Training in Minutes<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">A warm standby node is a backup node that is prepared in advance and kept on standby, so it can replace a failed node immediately without waiting for resources or rebuilding the environment. AI-Stack&#8217;s Automatic Training Recovery uses this approach and covers the three recovery requirements above in three steps:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Failure detection<\/strong>: The platform uses AlertManager to detect lost GPUs or devices in real time and triggers the recovery process.<\/li>\n\n\n\n<li><strong>Isolation and replacement<\/strong>: The platform automatically isolates the failed node and brings an idle, healthy node from the resource pool into the cluster.<\/li>\n\n\n\n<li><strong>Resume from checkpoint<\/strong>: Once the new node is initialized, the job resumes from the most recent checkpoint.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Note that the running job is still interrupted when a node fails. Automatic recovery saves the time spent requeuing, reacquiring resources, and rebuilding the cluster; it does not make the job immune to failures. Figure 2 compares the recovery paths of the three approaches.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img data-recalc-dims=\"1\" decoding=\"async\" width=\"1024\" height=\"420\" data-attachment-id=\"15788\" data-permalink=\"https:\/\/ai-stack.ai\/en\/ai-training-at-scale-slurm-on-k8s\/gpu-failure-recovery-checkpoint-auto-recovery-en-2\" data-orig-file=\"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/11\/9e1aa534-edited.png?fit=1600%2C657&amp;quality=100&amp;ct=202603031250&amp;ssl=1\" data-orig-size=\"1600,657\" data-comments-opened=\"0\" data-image-title=\"gpu-failure-recovery-checkpoint-auto-recovery-en\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/11\/9e1aa534-edited.png?fit=1024%2C420&amp;quality=100&amp;ct=202603031250&amp;ssl=1\" src=\"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/11\/9e1aa534-edited-1024x420.png?resize=1024%2C420&#038;quality=100&#038;ct=202603031250&#038;ssl=1\" alt=\"Comparison of three recovery paths after a GPU failure: no protection, checkpointing only, and checkpointing with Automatic Training Recovery\" class=\"wp-image-15788\" title=\"gpu-failure-recovery-checkpoint-auto-recovery-en.png\"\/><figcaption class=\"wp-element-caption\">Figure 2. Recovery paths of three approaches after a GPU failure<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Warm standby comes with two main costs:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Idle nodes must be reserved<\/strong>: These nodes do not take part in training, effectively trading some compute capacity for faster recovery. How many to reserve depends on cluster size and the cost of failures.<\/li>\n\n\n\n<li><strong>Progress since the last checkpoint must still be recomputed<\/strong>: Automatic recovery shortens recovery time but cannot reduce the progress lost between checkpoints, so checkpoint strategy still matters.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Checkpointing and Automatic Training Recovery must therefore work together: the former determines how much progress is lost, and the latter determines how quickly training resumes.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">AI-Stack Slurm on K8s Use Cases: Which Teams Benefit Most?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Not every AI team needs Slurm on K8s. Given the scheduling architecture and failure recovery mechanisms described above, the following four scenarios benefit the most.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Scenario 1: Academic and Supercomputing Centers with Existing Slurm Users<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Researchers at universities, research institutes, and national computing centers are often familiar with Slurm commands and have built up large collections of job scripts. These organizations want to upgrade bare-metal HPC clusters to a cloud-native architecture without making users learn new tools. Slurm on K8s lets researchers keep submitting jobs the way they always have, while a containerized platform takes over operations.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Scenario 2: Pre-Training and Continued Training of Large Language Models<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">LLM pre-training, continued pre-training, and large-scale fine-tuning typically require dozens of GPUs or more and run continuously for days or weeks. With the training framework pre-integrated, checkpointing combined with Automatic Training Recovery effectively reduces losses from interruptions during long training runs.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Scenario 3: Enterprise AI R&amp;D Centers with Multiple Teams Sharing GPUs<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">In enterprise AI R&amp;D centers across semiconductors, manufacturing, finance, and healthcare, a few teams typically need multi-node training while most work on inference services, development and testing, or small-scale training. Rather than building a dedicated cluster for large training jobs, organizations can let Slurm clusters share the same resource pool with other AI workloads and release resources for other uses once training is done.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Scenario 4: Workloads Highly Sensitive to the Cost of Training Interruptions<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The higher the hourly GPU cost and the longer a job runs, the greater the loss from a single interruption. For teams that have lost hours or even days of progress to a single GPU failure, checkpointing combined with Automatic Training Recovery delivers the most immediate value.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Recommended Hardware and Networking: Which Environments Benefit Most?<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">GPU Servers<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Multi-GPU servers across multiple nodes<\/strong>: A typical configuration uses servers with 8 GPUs each (for example, 8\u00d7 H200), combined into a training cluster. The more nodes, the greater the value of scheduling and failure recovery.<\/li>\n\n\n\n<li><strong>Reserved idle nodes<\/strong>: To enable Automatic Training Recovery, plan a number of spare idle nodes to serve as replacements when failures occur.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">High-Speed Networking<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Every step of multi-node training exchanges large amounts of data, so the network is often the deciding factor in training efficiency.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>InfiniBand or RoCE<\/strong>: Use InfiniBand or RDMA-capable RoCE high-speed networking between nodes. AI-Stack can enable high-performance networking when a cluster is created.<\/li>\n\n\n\n<li><strong>Validate with real measurements<\/strong>: Network specs do not equal real-world performance. After the cluster is created, measure cross-node bandwidth with NCCL Benchmark and confirm it meets targets before starting production training.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Storage<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Shared storage<\/strong>: Use shared storage such as GPFS or HDFS for mounting Jail (the Slurm user environment), checkpoints, training data, and workspaces.<\/li>\n\n\n\n<li><strong>Checkpoint write performance<\/strong>: Checkpoints for large models can reach hundreds of gigabytes, so storage write speed directly affects checkpoint frequency and training efficiency.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th><strong>Resource<\/strong><\/th><th><strong>Recommended configuration<\/strong><\/th><th><strong>Why it matters<\/strong><\/th><\/tr><\/thead><tbody><tr><td>GPU nodes<\/td><td>Multiple multi-GPU servers (e.g., 8 GPUs each), NVIDIA or AMD<\/td><td>Compute foundation for multi-node training<\/td><\/tr><tr><td>Standby nodes<\/td><td>Reserve idle nodes based on cluster size<\/td><td>Replacement source for Automatic Training Recovery<\/td><\/tr><tr><td>Networking<\/td><td>InfiniBand or RoCE<\/td><td>Determines inter-node communication efficiency<\/td><\/tr><tr><td>Storage<\/td><td>Shared storage such as GPFS or HDFS<\/td><td>Shared Jail environment, checkpoints, and training data<\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\">Table 2. Recommended hardware configuration for Slurm on K8s<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion: Five Questions to Ask Before Adopting Slurm on K8s<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">In the architecture described here, Slurm handles resource scheduling, Kubernetes handles operations, and checkpointing together with Automatic Training Recovery strengthens fault resilience. So when evaluating a multi-node training platform, rather than asking \u201cWhich scheduler should we choose?\u201d, enterprises should ask a more fundamental question:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Given our current user workflows, GPU scale, and tolerance for lost progress, what scheduling and operations architecture do we need, and how much capacity are we willing to reserve for failure recovery?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Before adopting or adjusting your architecture, assess where you stand in these five areas:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>User workflows<\/strong>: Does your organization already have Slurm users and existing job scripts? This determines how valuable it is to keep Slurm.<\/li>\n\n\n\n<li><strong>Resource sharing<\/strong>: Do training, inference, and development need to share the same GPU pool, or is a dedicated cluster a better fit?<\/li>\n\n\n\n<li><strong>Network validation<\/strong>: Has your cross-node network been tested with tools such as NCCL Benchmark, rather than judged by its spec sheet alone?<\/li>\n\n\n\n<li><strong>Checkpoint strategy<\/strong>: Do your checkpoint interval and storage performance match the progress loss you can accept? Have you actually practiced restoring from a checkpoint?<\/li>\n\n\n\n<li><strong>Redundancy cost<\/strong>: How much does a single failure cost? How many idle nodes are worth reserving?<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Technology will keep evolving and clusters will keep growing, but as long as GPUs can fail and resources must be allocated, these five questions are worth revisiting regularly.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Frequently Asked Questions (FAQ)<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Q: How is Slurm on K8s different from using Kubernetes directly?<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The Kubernetes default scheduler assigns nodes to Pods one at a time, which can cause resource deadlocks in distributed training, and it cannot reuse Slurm commands or existing job scripts. With Slurm on K8s, Slurm handles job scheduling and preserves the workflow HPC users already know, while Kubernetes handles containerized deployment and operations.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Q: What is gang scheduling?<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Gang scheduling treats all of a job&#8217;s Pods as a single unit and starts them together only when all required resources are available. This prevents deadlocks in which multiple distributed jobs each hold some GPUs while waiting on one another. Native gang scheduling in Kubernetes reached beta in v1.37 but is not enabled by default.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Q: How long does it take to recover from a GPU failure during distributed training?<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">With checkpointing alone, the entire job must be requeued and reinitialized after a failure, which often takes hours. With AI-Stack&#8217;s Automatic Training Recovery, failed nodes are automatically replaced by warm standby nodes, cutting recovery time to minutes. Either way, progress since the last checkpoint must be recomputed.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Q: How should checkpoint frequency be set?<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The optimal checkpoint interval depends on the cost of writing a checkpoint and the failure rate. Checkpointing too often eats into training time, while checkpointing too rarely means losing more progress with each failure. In general, the more nodes and the more frequent the failures, the shorter the interval should be.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Q: What hardware does AI-Stack Slurm on K8s require?<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Recommended configurations include multiple multi-GPU servers (NVIDIA or AMD), InfiniBand or RoCE high-speed networking, and shared storage such as GPFS or HDFS. To enable Automatic Training Recovery, you also need to reserve idle nodes as replacements.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>AI training at scale faces three challenges: scheduling, operations, and GPU failures. See how AI-Stack Slurm on K8s solves them and recovers training in minutes.<\/p>\n","protected":false},"author":253372381,"featured_media":15783,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[96987570,96987598],"tags":[96987654,96988178,96989085],"class_list":["post-15773","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technical-support","category-product-features","tag-ai-en","tag-hpc-en","tag-slurm"],"blocksy_meta":[],"acf":[],"jetpack_shortlink":"https:\/\/wp.me\/ph344V-46p","jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/11\/7310d13d.jpg?fit=1920%2C1080&quality=100&ct=202603031250&ssl=1","_links":{"self":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts\/15773","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/users\/253372381"}],"replies":[{"embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/comments?post=15773"}],"version-history":[{"count":2,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts\/15773\/revisions"}],"predecessor-version":[{"id":15792,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts\/15773\/revisions\/15792"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/media\/15783"}],"wp:attachment":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/media?parent=15773"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/categories?post=15773"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/tags?post=15773"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}