Is spinning up an LLM really as simple as downloading the model? Not quite. GPU allocation, choosing the right inference engine, model storage — even Hugging Face’s licensing and access tokens — can all become roadblocks during deployment.

Infinitix’s AI-Stack platform helps enterprises centrally orchestrate GPU compute and manage AI infrastructure. Its Rapid Container Service (RCS) lets teams deploy models and application services quickly, without having to rebuild their environment from scratch.

In this post, we’ll walk through, step by step, how to use RCS on AI-Stack together with vLLM to deploy the google/gemma-3-1b-it language model. We’ll also cover a scenario you’re likely to run into with gated models on Hugging Face — how to complete the authorization and token setup.

Hands-On: Deploying vLLM with RCS to Get Gemma-3-1B Running

Set Up a Storage Cluster in the Admin Console

 First, to avoid re-downloading the model every time you deploy, we need to configure a storage cluster to hold the model files. Log in to the AI-Stack admin console, go to Settings > Storage Cluster > Storage Cluster Management, and create a new storage cluster.

Once the storage cluster is created, confirm its status shows Active. You can then create a storage volume on this cluster for use in the upcoming vLLM deployment.

Next, log in to the AI-Stack user console. From the left-hand menu, go to Machine Learning Projects > Project List, and select the project you want to deploy the model to. In this example, we’ll use Project 1.

Create an RCS Container

Next, head to Rapid Container Service > Deploy Application and configure the following:

  1. Name: custom — we’ll use demo for this example.
  2. Revision History Limit: how many previous versions to keep, so you can roll back later if needed.
  3. Replica Count: the number of container replicas running simultaneously to serve requests. More replicas support higher concurrency; since this is just a demo, we’ll set it to 1.

Once configured, click Add Container.

Now let’s add the container. Click Add Container, give it a Name, then click Select Image.

On the image selection screen, choose Manual Download, and enter vLLM’s official image under Image Path. This example uses: vllm/vllm-openai:v0.26.0-cu129-ubuntu2404

With the image selected, return to the container configuration page and set the following resources:

  1. Specs: choose GPU resources based on the model size. Start by estimating the GPU memory the model itself needs, then leave some headroom for vLLM’s runtime overhead. For example, if a model needs about 6 GB of GPU memory, an 8–9 GB GPU spec gives you a reasonable buffer.
  2. Port: enter the default port, 8000. vLLM serves its API on this port by default.

Get Access to the Hugging Face Model

Here’s where this deployment gets a little more interesting. Since gemma-3-1b-it is a gated model, you’ll need to request access first. Log in to your Hugging Face account, open a new tab, and go to the model page you want to deploy — in this case, huggingface.co/google/gemma-3-1b-it — then click Acknowledge license to complete the authorization.

This step matters — only after accepting the model’s license will your account be granted access to it.

Figure 1: Enter the URL of the model you want to deploy

Once you have access, you’ll need to generate a Hugging Face token so the vLLM container can authenticate and download the model. Go to https://huggingface.co/settings/tokens and click Create new token.

There are two key settings when creating the token:

  • Token name: custom — we’ll use demo for this example.
  • Token Type: if you only need vLLM to download the model, Read access is sufficient.

Once set, click Create Token, and the system will generate a token starting with hf_.

One important note: the full token is only ever shown once, at creation. Be sure to click the copy button right away.

With the token copied, switch back to the AI-Stack container configuration page, find Environment Variables, and add a new one:

  • Key: HF_TOKEN (the default environment variable name Hugging Face’s library looks for)
  • Value: paste the hf_-prefixed token you just copied

Mount the Model Storage Volume

Next, click Mount > Mount Volume.

Select the storage volume you created earlier. Under Mount Path, enter: /root/.cache/huggingface

Mounting the Hugging Face cache directory to your storage volume means downloaded models persist between sessions — so the next time you create or start a container, you won’t need to download the model again.

Set the vLLM Startup Command

Back on the Add Container page, find Command, and enter:

vllm serve –model google/gemma-3-1b-it –served-model-name gemma3-1b –gpu-memory-utilization 0.7 –enforce-eager

This command launches vLLM and loads the specified model. Here’s what each flag does:

  • –model google/gemma-3-1b-it: specifies the model to load — google/gemma-3-1b-it is the model’s name on Hugging Face.
  • –served-model-name gemma3-1b: sets the model name exposed via the API, so you can reference it as gemma3-1b when calling the API.
  • –gpu-memory-utilization 0.95: sets the fraction of GPU memory vLLM is allowed to use — 95% in this example.

Configure Shared Memory

Next, under Shared Memory, decide whether to enable it based on your model and inference workload. Since google/gemma-3-1b-it is a relatively small model, no special configuration is needed here; larger models may require additional shared memory depending on your workload.

Once everything is set, click Deploy on the right to launch the vLLM container.

Confirm vLLM Has Started Successfully

After deployment, click Pod in the left-hand menu and wait for its status to change to Running. Once the pod is running, select it, go to Containers, check the corresponding container, and click Logs.

In the log view, you can watch vLLM’s startup progress in real time. Keep refreshing until you see Application startup complete.

That means vLLM has started successfully and the model has finished loading.

Configure the Service

Once the vLLM container is running, you still need to expose a service endpoint so other applications can call the model. Next, head to Services to set up an external API endpoint.

Go to the Services page, click Add Service, and configure the following:

  1. Name: custom — we recommend matching the container name you deployed earlier, so it’s easy to track.
  2. Type: select NodePort.
  3. Port: enter 8000, vLLM’s default port for serving its API.
  4. Protocol: select TCP.

Once configured, click Add to finish creating the service.

After the service is created, click into it to find the corresponding Endpoint URL. Copy this URL, and you can use it to connect your Agent application — putting the Gemma-3-1B model into production for inference.

And that’s how you deploy a model on AI-Stack using RCS! Along the way, we covered how to set up a vLLM container, configure GPU and storage resources, and walked through the authorization and token setup for a gated model. With AI-Stack’s RCS function, developers get the flexibility to deploy the LLMs they need and quickly spin up a dedicated AI inference service of their own.