{"id":10592,"date":"2025-07-28T16:19:32","date_gmt":"2025-07-28T08:19:32","guid":{"rendered":"https:\/\/ai-stack.ai\/kubeflow-ixgpu-tutorial"},"modified":"2025-07-28T17:50:52","modified_gmt":"2025-07-28T09:50:52","slug":"kubeflow-ixgpu-tutorial","status":"publish","type":"post","link":"https:\/\/ai-stack.ai\/en\/kubeflow-ixgpu-tutorial","title":{"rendered":"Master Kubeflow GPU Partitioning: Hands-on with INFINITIX ixGPU Module for High-Efficiency Resource Utilization!"},"content":{"rendered":"\n

Kubeflow<\/a>, an open-source platform built on Kubernetes<\/a>, has become widely used by many data scientists and machine learning engineers. This is because it provides a standardized and integrated set of tools covering various stages, including data preparation, model training, hyperparameter tuning, model deployment, and monitoring, significantly simplifying the process of bringing AI models into production.<\/p>\n\n\n\n

However, many developers still encounter the problem of being unable to partition GPU resources when using Kubeflow. Due to its inherent characteristics, a single container will occupy an entire GPU’s resources. When a project requires less compute power, it leads to some GPU capacity sitting idle and not being effectively utilized. Today’s article will provide a hands-on guide on how to use INFINITIX’s ixGPU module to perform GPU partitioning on the Kubeflow platform.<\/p>\n\n\n\n

Leveraging the ixGPU Module on the Kubeflow Platform for Flexible GPU Partitioning<\/h2>\n\n\n\n

Custom YAML PodDefault Configuration<\/h3>\n\n\n\n

We’re using a Tesla P4 GPU<\/strong> for this GPU partitioning demonstration. Before applying the ixGPU module to Kubeflow, you’ll need to prepare your YAML files. We’ve set up two YAML files for this: poddefault1.yaml and poddefault2.yaml.<\/p>\n\n\n\n

As shown in the image, by running the following two commands, we can see the detailed configurations of the PodDefault resources defined within these two YAML files: <\/p>\n\n\n\n

cat poddefault1.yaml <\/code><\/pre>\n\n\n\n
cat poddefault2.yaml<\/code><\/pre>\n\n\n\n
These two YAML files define distinct PodDefault configurations. They’re designed to allocate different sizes of Tesla P4 GPU memory resources\u2014specifically, 1GB and 3GB\u2014to Pods within the Kubeflow environment that are tagged with specific labels.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Next, you’ll use the kubectl apply command to read the defined PodDefault configurations (which contain the GPU partitioning information) from the YAML files and apply them to your Kubernetes cluster.<\/p>\n\n\n\n
kubectl apply -f poddefault1.yaml<\/code><\/pre>\n\n\n\n
kubectl apply -f poddefault2.yaml<\/code><\/pre>\n\n\n\n
Afterward, run the kubectl get poddefault -A command to query detailed information about all PodDefault resources across all namespaces in your current Kubernetes cluster.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Kubeflow Notebooks GPU Partitioning<\/h3>\n\n\n\n
Once the preparatory steps are complete, we’ll head over to the Kubeflow platform to create a new notebook container.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Click on Advanced Options<\/strong>, and under the Configurations<\/strong> section, you’ll see the two new GPU resource specifications we just added: Tesla P4 1GB<\/strong> and Tesla P4 3GB<\/strong>.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Once both containers are created (notebook1 with 1GB and notebook2 with 3GB), we can check if the container resources have truly been partitioned. Click CONNECT<\/strong> to open notebook1’s Jupyter Lab.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Next, we can use the !nvidia-smi command to verify the current resources within the container. As shown in the image, the memory resource in notebook1 is 1GB.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Follow the same steps for notebook2: click CONNECT<\/strong> to launch Jupyter Lab.<\/p>\n\n\n\n
After entering the !nvidia-smi command, you’ll see that this container has 3GB of memory resources.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
There you have it! Flexible GPU partitioning on Kubeflow is complete! Pretty straightforward, right?<\/p>\n\n\n\n
Next up, we’ll walk you through how to apply the ixGPU module within Kubeflow Pipelines. This will let us allocate the exact GPU memory resources needed for different tasks throughout your entire workflow, achieving even more precise and efficient flexible allocation to maximize your compute effectiveness.<\/p>\n\n\n\n
Integrate the ixGPU Module into Kubeflow Pipeline for High-Efficiency GPU Resource Allocation<\/h2>\n\n\n\n
In this example, we’ve set up a simple Pipeline with two stages: Training and Testing.<\/p>\n\n\n\n
Machine Learning Pipeline GPU Resource Allocation Pre-configuration<\/h3>\n\n\n\n
In pipeline.py, the Python script used to define our Kubeflow Pipeline, we first define the GPU resource allocation for the Training<\/strong> and Testing<\/strong> steps. Since Training has higher compute demands, we’ve allocated 3GB of GPU memory<\/strong>; for Testing, which requires less compute, we’ve allocated 1GB of resources<\/strong>. Please refer to the image below for the code.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Next, we’ll enter python pipeline.py in the TERMINAL to compile the entire workflow. After execution, you’ll see mnist-pipeline.yaml appear in the left-hand list. This indicates that our Python script has been successfully run, and the entire machine learning workflow we defined in the Python code has been compiled into a YAML file that conforms to Kubeflow Pipeline specifications.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Kubeflow Platform Pipeline Configuration<\/h3>\n\n\n\n
Next, head over to the Pipelines page on Kubeflow and click + Upload pipeline in the top-right corner to create a new pipeline version.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Select the “Upload a file”<\/strong> option and upload the mnist-pipeline.yaml file you just compiled. This is the crucial step for deploying the machine learning workflow, which was developed and defined locally, to the Kubeflow cluster for execution and management.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
After a successful creation, the Kubeflow Pipelines user interface will display an overview page for the newly uploaded mnist-pipeline. You can then click + Create run to start executing your machine learning Pipeline.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Once the training process is complete, you can click in to view the logs.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
You can see that in this machine learning model training workflow, we successfully completed the training task using 3GB of GPU memory resources.<\/p>\n\n\n\n
Next, you can check the execution logs for the testing task.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
We successfully partitioned the GPU and allocated 1GB of resources to the testing task.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
That wraps up our complete tutorial on applying the ixGPU module within the Kubeflow platform!<\/p>\n\n\n\n
INFINITIX’s ixGPU module can help enterprises utilize GPU compute resources more flexibly. After seeing the GPU partitioning capabilities of the ixGPU module combined with Kubeflow, are you also interested in this feature? Feel free to contact us to learn more! <\/p>\n\n\n\n
To better understand the core advantages of the ixGPU module, please refer to this article: Breaking Through Kubeflow Limitations: INFINITIX ixGPU Module Achieves Flexible GPU Partitioning<\/a><\/strong><\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
Many developers still encounter issues with GPU resource partitioning when using Kubeflow. This article will guide you step-by-step on how to perform Kubeflow GPU partitioning using Infinitix’s ixGPU module.<\/p>\n","protected":false},"author":253372381,"featured_media":10593,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_crdt_document":"","jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[96987598,96987570],"tags":[96988171,96988172,96988174],"class_list":["post-10592","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-product-features","category-technical-support","tag-ixgpu-en","tag-kubeflow-en","tag-gpu-partitioning"],"blocksy_meta":[],"acf":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2025\/07\/AI%E8%B3%87%E6%96%99%E4%B8%AD%E5%BF%83%E6%98%AF%E4%BB%80%E9%BA%BC-%E8%B7%9F%E5%82%B3%E7%B5%B1%E8%B3%87%E6%96%99%E4%B8%AD%E5%BF%83%E6%9C%89%E4%BB%80%E9%BA%BC%E5%B7%AE%E5%88%A5-5.png?fit=1920%2C1080&quality=100&ct=202603031250&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/ph344V-2KQ","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts\/10592","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/users\/253372381"}],"replies":[{"embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/comments?post=10592"}],"version-history":[{"count":0,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts\/10592\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/media\/10593"}],"wp:attachment":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/media?parent=10592"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/categories?post=10592"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/tags?post=10592"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}