{"id":15097,"date":"2026-09-14T16:27:20","date_gmt":"2026-09-14T08:27:20","guid":{"rendered":"https:\/\/ai-stack.ai\/rcs-vllm-deployment"},"modified":"2026-09-14T17:23:07","modified_gmt":"2026-09-14T09:23:07","slug":"rcs-vllm-deployment","status":"publish","type":"post","link":"https:\/\/ai-stack.ai\/en\/rcs-vllm-deployment","title":{"rendered":"How to Deploy a Large Language Model Service on AI-Stack Using RCS and the vLLM Framework?\u00a0"},"content":{"rendered":"\n

Is spinning up an LLM really as simple as downloading the model? Not quite. GPU allocation, choosing the right inference engine, model storage \u2014 even Hugging Face’s licensing and access tokens \u2014 can all become roadblocks during deployment.<\/p>\n\n\n\n

Infinitix’s AI-Stack<\/a> platform helps enterprises centrally orchestrate GPU compute and manage AI infrastructure. Its Rapid Container Service (RCS<\/strong>) lets teams deploy models and application services quickly, without having to rebuild their environment from scratch.<\/p>\n\n\n\n

In this post, we’ll walk through, step by step, how to use RCS on AI-Stack together with vLLM to deploy the google\/gemma-3-1b-it language model. We’ll also cover a scenario you’re likely to run into with gated models on Hugging Face<\/a> \u2014 how to complete the authorization and token setup.<\/p>\n\n\n\n

Hands-On: Deploying vLLM with RCS to Get Gemma-3-1B Running<\/strong><\/h2>\n\n\n\n

Set Up a Storage Cluster in the Admin Console<\/strong><\/h3>\n\n\n\n

\u00a0First, to avoid re-downloading the model every time you deploy, we need to configure a storage cluster to hold the model files. Log in to the AI-Stack admin console<\/strong>, go to Settings > Storage Cluster > Storage Cluster Management<\/strong>, and create a new storage cluster.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Once the storage cluster is created, confirm its status shows Active<\/strong>. You can then create a storage volume on this cluster for use in the upcoming vLLM deployment.<\/p>\n\n\n\n

Next, log in to the AI-Stack user console<\/strong>. From the left-hand menu, go to Machine Learning Projects > Project List<\/strong>, and select the project you want to deploy the model to. In this example, we’ll use Project 1<\/strong>.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Create an RCS Container<\/strong><\/h3>\n\n\n\n

Next, head to Rapid Container Service > Deploy Application<\/strong> and configure the following:<\/p>\n\n\n\n

    \n
  1. Name<\/strong>: custom \u2014 we’ll use demo for this example.<\/li>\n\n\n\n
  2. Revision History Limit<\/strong>: how many previous versions to keep, so you can roll back later if needed.<\/li>\n\n\n\n
  3. Replica Count<\/strong>: the number of container replicas running simultaneously to serve requests. More replicas support higher concurrency; since this is just a demo, we’ll set it to 1.<\/li>\n<\/ol>\n\n\n\n

    Once configured, click Add Container<\/strong>.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    <\/p>\n\n\n\n

    Now let’s add the container. Click Add Container<\/strong>, give it a Name<\/strong>, then click Select Image<\/strong>.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    On the image selection screen, choose Manual Download<\/strong>, and enter vLLM’s official image under Image Path<\/strong>. This example uses: vllm\/vllm-openai:v0.26.0-cu129-ubuntu2404<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    With the image selected, return to the container configuration page and set the following resources:<\/p>\n\n\n\n

      \n
    1. Specs<\/strong>: choose GPU resources based on the model size. Start by estimating the GPU memory the model itself needs, then leave some headroom for vLLM’s runtime overhead. For example, if a model needs about 6 GB of GPU memory, an 8\u20139 GB GPU spec gives you a reasonable buffer.<\/li>\n\n\n\n
    2. Port<\/strong>: enter the default port, 8000. vLLM serves its API on this port by default.<\/li>\n<\/ol>\n\n\n\n
      \"\"<\/figure>\n\n\n\n

      Get Access to the Hugging Face Model<\/strong><\/h3>\n\n\n\n

      Here’s where this deployment gets a little more interesting. Since gemma-3-1b-it is a gated model, you’ll need to request access first. Log in to your Hugging Face account, open a new tab, and go to the model page you want to deploy \u2014 in this case, huggingface.co\/google\/gemma-3-1b-it<\/a> \u2014 then click Acknowledge license<\/strong> to complete the authorization.<\/p>\n\n\n\n

      This step matters \u2014 only after accepting the model’s license will your account be granted access to it.<\/p>\n\n\n\n

      \"\"
      Figure 1: Enter the URL of the model you want to deploy<\/em>
      <\/figcaption><\/figure>\n\n\n\n

      <\/p>\n\n\n\n

      \"\"<\/figure>\n\n\n\n

      Once you have access, you’ll need to generate a Hugging Face token so the vLLM container can authenticate and download the model. Go to https:\/\/huggingface.co\/settings\/tokens and click Create new token<\/strong>.<\/p>\n\n\n\n

      \"\"<\/figure>\n\n\n\n

      There are two key settings when creating the token:<\/p>\n\n\n\n

        \n
      • Token name<\/strong>: custom \u2014 we’ll use demo for this example.<\/li>\n\n\n\n
      • Token Type<\/strong>: if you only need vLLM to download the model, Read<\/strong> access is sufficient.<\/li>\n<\/ul>\n\n\n\n

        Once set, click Create Token<\/strong>, and the system will generate a token starting with hf_.<\/p>\n\n\n\n

        \"\"<\/figure>\n\n\n\n

        <\/p>\n\n\n\n

        One important note: the full token is only ever shown once, at creation. Be sure to click the copy button right away.
        <\/p>\n\n\n\n

        \"\"<\/figure>\n\n\n\n

        With the token copied, switch back to the AI-Stack container configuration page, find Environment Variables<\/strong>, and add a new one:<\/p>\n\n\n\n

          \n
        • Key<\/strong>: HF_TOKEN (the default environment variable name Hugging Face’s library looks for)<\/li>\n\n\n\n
        • Value<\/strong>: paste the hf_-prefixed token you just copied<\/li>\n<\/ul>\n\n\n\n
          \"\"<\/figure>\n\n\n\n

          Mount the Model Storage Volume<\/strong><\/h3>\n\n\n\n

          Next, click Mount > Mount Volume<\/strong>. <\/p>\n\n\n\n

          \"\"<\/figure>\n\n\n\n

          Select the storage volume you created earlier. Under Mount Path<\/strong>, enter: \/root\/.cache\/huggingface<\/p>\n\n\n\n

          \"\"<\/figure>\n\n\n\n

          Mounting the Hugging Face cache directory to your storage volume means downloaded models persist between sessions \u2014 so the next time you create or start a container, you won’t need to download the model again.<\/p>\n\n\n\n

          Set the vLLM Startup Command<\/strong><\/h3>\n\n\n\n

          Back on the Add Container<\/strong> page, find Command<\/strong>, and enter:
          <\/p>\n\n\n\n

          vllm serve –model google\/gemma-3-1b-it –served-model-name gemma3-1b –gpu-memory-utilization 0.7 –enforce-eager<\/p>\n\n\n\n

          This command launches vLLM and loads the specified model. Here’s what each flag does:<\/p>\n\n\n\n

            \n
          • –model google\/gemma-3-1b-it: specifies the model to load \u2014 google\/gemma-3-1b-it is the model’s name on Hugging Face.<\/li>\n\n\n\n
          • –served-model-name gemma3-1b: sets the model name exposed via the API, so you can reference it as gemma3-1b when calling the API.<\/li>\n\n\n\n
          • –gpu-memory-utilization 0.95: sets the fraction of GPU memory vLLM is allowed to use \u2014 95% in this example.<\/li>\n<\/ul>\n\n\n\n
            \"\"<\/figure>\n\n\n\n

            Configure Shared Memory<\/strong><\/h3>\n\n\n\n

            Next, under Shared Memory<\/strong>, decide whether to enable it based on your model and inference workload. Since google\/gemma-3-1b-it is a relatively small model, no special configuration is needed here; larger models may require additional shared memory depending on your workload.<\/p>\n\n\n\n

            \"\"<\/figure>\n\n\n\n

            Once everything is set, click Deploy<\/strong> on the right to launch the vLLM container.
            <\/p>\n\n\n\n

            \"\"<\/figure>\n\n\n\n

            Confirm vLLM Has Started Successfully<\/strong><\/h3>\n\n\n\n

            After deployment, click Pod<\/strong> in the left-hand menu and wait for its status to change to Running<\/strong>. Once the pod is running, select it, go to Containers<\/strong>, check the corresponding container, and click Logs<\/strong>.<\/p>\n\n\n\n

            \"\"<\/figure>\n\n\n\n

            In the log view, you can watch vLLM’s startup progress in real time. Keep refreshing until you see Application startup complete.<\/strong><\/p>\n\n\n\n

            That means vLLM has started successfully and the model has finished loading.<\/p>\n\n\n\n

            \"\"<\/figure>\n\n\n\n

            Configure the Service<\/strong><\/h3>\n\n\n\n

            Once the vLLM container is running, you still need to expose a service endpoint so other applications can call the model. Next, head to Services<\/strong> to set up an external API endpoint.<\/p>\n\n\n\n

            Go to the Services<\/strong> page, click Add Service<\/strong>, and configure the following:<\/p>\n\n\n\n

              \n
            1. Name<\/strong>: custom \u2014 we recommend matching the container name you deployed earlier, so it’s easy to track.<\/li>\n\n\n\n
            2. Type<\/strong>: select NodePort<\/strong>.<\/li>\n\n\n\n
            3. Port<\/strong>: enter 8000, vLLM’s default port for serving its API.<\/li>\n\n\n\n
            4. Protocol<\/strong>: select TCP<\/strong>.<\/li>\n<\/ol>\n\n\n\n

              Once configured, click Add<\/strong> to finish creating the service.<\/p>\n\n\n\n

              \"\"<\/figure>\n\n\n\n

              After the service is created, click into it to find the corresponding Endpoint<\/strong> URL. Copy this URL, and you can use it to connect your Agent application \u2014 putting the Gemma-3-1B model into production for inference.<\/p>\n\n\n\n

              \"\"<\/figure>\n\n\n\n

              And that’s how you deploy a model on AI-Stack using RCS! Along the way, we covered how to set up a vLLM container, configure GPU and storage resources, and walked through the authorization and token setup for a gated model. With AI-Stack’s RCS function, developers get the flexibility to deploy the LLMs they need and quickly spin up a dedicated AI inference service of their own.<\/p>\n\n\n\n

              <\/p>\n","protected":false},"excerpt":{"rendered":"

              In this post, we’ll walk through, step by step, how to use RCS on AI-Stack together with vLLM to deploy the google\/gemma-3-1b-it language model.<\/p>\n","protected":false},"author":253372381,"featured_media":15099,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[96987598,96987570],"tags":[96989042,96989043],"class_list":["post-15097","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-product-features","category-technical-support","tag-rcs","tag-vllm"],"blocksy_meta":[],"acf":[],"jetpack_shortlink":"https:\/\/wp.me\/ph344V-3Vv","jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/09\/f1a32512.png?fit=1920%2C1080&quality=100&ct=202603031250&ssl=1","_links":{"self":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts\/15097","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/users\/253372381"}],"replies":[{"embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/comments?post=15097"}],"version-history":[{"count":2,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts\/15097\/revisions"}],"predecessor-version":[{"id":15104,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts\/15097\/revisions\/15104"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/media\/15099"}],"wp:attachment":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/media?parent=15097"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/categories?post=15097"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/tags?post=15097"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}