S
SIEmon
Guest
Serving open-source models on Kubernetes requires much more than one tool. Ha! It’s more like a full-fledged orchestra. You need a model runner, model deployment tools, cluster management, and a few other components to get everything working together on your GPU.
The best choice for each of these jobs depends on what you’re serving. Is it one large model? Or many small ones?
In this quick-to-digest guide (a shoutout to our short attention spans
), I will walk you through each job, the main open-source tools for it, and the trade-offs you should consider.
If you only need a ranking of inference servers, I’ve already covered that in Top Kubernetes-Native Inference Servers Ranked (2026).
Most rankings put all the tools on one list, but that makes them hard to compare because they do different jobs:
Every platform in this guide eventually calls a runtime. For open-source large language models (LLMs), there are three options you should know about:
Do you need a custom resource definition (CRD) for one model or a full platform for many?
A plain Kubernetes Service distributes requests evenly across pods, but it doesn't know which pod already has a prompt cached or which one has a full queue.
GPUs cost money even when they’re not processing requests:
Scale-to-zero comes with a hidden cost: cold starts. SIE's autoscaling docs break the whole process down into 2-5 minutes to find a GPU node, 20-40 seconds to start containers, and 10-120 seconds to load the model. That’s 3-7 minutes in total. Any platform waiting on a fresh GPU node faces similar delays. (ಠ_ಠ)
So which combination should you deploy?
Not sure which option to choose? Superlinked's llm-d vs Dynamo vs SIE comparison suggests a simple test: count how many distinct models one user request touches.
If the table points you to the small-model row, that’s the part I wouldn’t build from scratch.
The do-it-yourself version means one deployment per embedding model, reranker, and extractor, one autoscaler config each, a gateway in front of all of them, and GPUs that spend much of their time idle between requests.
SIE puts all that behind one API with four operations:
In Superlinked's write-up from the Infer() Summit, CEO Daniel Svonava reported more than 50% higher throughput on real small-model workloads from SIE's shared queue, which forms a full batch before the GPU runs.
To be clear about the limits, though:
If this sounds like a fit, the quickstart takes a few minutes. For more depth, read How to Serve a Fleet of Small Open-Source Models, How to Choose an Inference Layer for Agents, and Hundreds of Models, One Deployment.
Happy building!
There’s really no single best tool. vLLM or SGLang is usually the runtime, and the platform depends on your workload: that is, whether you serve one large model (llm-d, KServe, Dynamo) or many small ones (SIE).
TGI has been in maintenance mode since December 2025, so new deployments should start on vLLM or SGLang.
Yes, with KEDA or Knative, but expect cold starts of several minutes when a new GPU node needs to start.
Yes, they run as separate control planes on separate GPU pools and communicate over HTTP.
The best choice for each of these jobs depends on what you’re serving. Is it one large model? Or many small ones?
In this quick-to-digest guide (a shoutout to our short attention spans
), I will walk you through each job, the main open-source tools for it, and the trade-offs you should consider.If you only need a ranking of inference servers, I’ve already covered that in Top Kubernetes-Native Inference Servers Ranked (2026).
5 Layers of a Kubernetes Serving Stack
Most rankings put all the tools on one list, but that makes them hard to compare because they do different jobs:
| Layer | What it does | Tools in this guide |
|---|---|---|
| Runtime | Loads model weights onto the GPU and turns requests into outputs | vLLM, SGLang, TGI |
| Serving platform | Deploys runtimes as Kubernetes resources and handles model config, rollouts, and scaling rules | KServe, llm-d, NVIDIA Dynamo, KAITO, Ray Serve, SIE |
| Routing | Sends each request to the pod best placed to answer it | Gateway API Inference Extension, vLLM Production Stack router |
| Autoscaling | Adds and removes GPU pods as traffic changes | KEDA, Knative, Horizontal Pod Autoscaler (HPA) |
| GPU setup | Makes GPUs visible and schedulable in the cluster | NVIDIA device plugin, NVIDIA GPU Operator |
Runtimes: vLLM vs. SGLang vs. TGI
Every platform in this guide eventually calls a runtime. For open-source large language models (LLMs), there are three options you should know about:
| Runtime | Best for | Trade-offs | Status |
|---|---|---|---|
| vLLM | The default for generative LLMs: wide model support and an OpenAI-compatible API. | Built around one model per server. Small encoders don’t benefit much from its LLM features. | Active Used by llm-d, KServe, and KAITO |
| SGLang | Requests that share long prompt prefixes (agents, multi-turn chat) and structured output. | Most Kubernetes operators default to vLLM, so support varies by platform. | ActiveBackend for Dynamo and SIE |
| TGI | Existing deployments you cannot migrate yet. | Minor fixes only, no new features. | Maintenance mode since December 2025 |
If you still run TGI: Hugging Face now accepts only bug fixes and documentation updates and recommends vLLM, SGLang, llama.cpp, or MLX going forward.
Runtime Checklist
- Pick vLLM if you want the safest default.
- Test SGLang if many requests start with the same long system prompt.
- Schedule a migration if anything in production still runs TGI.
- Measure embedding or reranking models before putting them on an LLM runtime (SIE vs. vLLM shows why the results often end up being a legit Kinder Surprise).
Which Serving Platform Should Manage Your Models?
Do you need a custom resource definition (CRD) for one model or a full platform for many?
| Platform | Best for | Trade-offs | License |
|---|---|---|---|
| KServe | Standard serving for both predictive and LLM workloads | Many moving parts; a rough start for small teams | Apache 2.0 (CNCF Incubating) |
| llm-d | One or a few large models at high concurrency | Generation only; its main benefits require large models and fast interconnects | Apache 2.0 (CNCF Sandbox) |
| NVIDIA Dynamo | Multi-node LLM serving on NVIDIA GPUs | Extensive configuration options; built around NVIDIA hardware | Apache 2.0 |
| KAITO | Going from a Hugging Face model to an endpoint quickly, especially on Azure | Runs vLLM and Hugging Face models only | Apache 2.0 (CNCF Sandbox) |
| Ray Serve (via KubeRay) | Multi-step pipelines you want to write in Python | A Ray cluster brings its own operations work | Apache 2.0 |
| SIE | An agent's fleet of small models sharing a few GPUs | Not built for one giant model; scaling from zero takes minutes | Apache 2.0 |
How Do Requests Reach the Right Pod?
A plain Kubernetes Service distributes requests evenly across pods, but it doesn't know which pod already has a prompt cached or which one has a full queue.
| Tool | What it adds | Best for | Trade-offs |
|---|---|---|---|
| Gateway API Inference Extension | An InferencePool resource; your gateway asks an Endpoint Picker (EPP) which pod should take each request | Teams already on Envoy Gateway, kgateway, GKE Gateway, or agentgateway | Since v1.6.0, the full Endpoint Picker is in the llm-d project |
| vLLM Production Stack router | A Helm-installed router in front of vLLM pods, with service discovery, fault tolerance, and a monitoring dashboard | Clusters that run only vLLM and want a reference setup | vLLM only |
| Built-in routers (llm-d, Dynamo, SIE gateway) | Routing optimized for a specific platform | Teams that have already chosen a particular platform | Tied to that platform |
Routing Checklist
- Are you running one platform? Use its built-in router first.
- Want to mix gateways or vendors? Standardize on
InferencePool.
Scaling GPU Pods Without Paying for Idling
GPUs cost money even when they’re not processing requests:
| Autoscaler | Scales on | Scale to zero | Best for |
|---|---|---|---|
| HPA | CPU, memory, or custom metrics | Not by default | Steady traffic where one warm pod is always worth the cost |
| KEDA | Prometheus metrics such as queue depth or pending requests | Yes | GPU workloads; KAITO and SIE use it, and KServe supports it |
| Knative | Concurrent requests per pod | Yes | KServe's serverless mode and HTTP-driven traffic |
Scale-to-zero comes with a hidden cost: cold starts. SIE's autoscaling docs break the whole process down into 2-5 minutes to find a GPU node, 20-40 seconds to start containers, and 10-120 seconds to load the model. That’s 3-7 minutes in total. Any platform waiting on a fresh GPU node faces similar delays. (ಠ_ಠ)
Autoscaling Checklist
- Keep at least one warm replica for anything users wait on.
- Use scale-to-zero for batch jobs and internal tools.
- Set a cooldown that matches your traffic patterns (SIE defaults to 15 minutes before scaling down).
Which Stack Is the Best Fit for Your Workload?
So which combination should you deploy?
| If you are building... | Runtime | Platform | Routing | Autoscaling |
|---|---|---|---|---|
| One large chat model at high concurrency | vLLM | llm-d or KServe LLMInferenceService | Inference Extension + llm-d EPP | llm-d autoscaler or KEDA |
| Multi-node serving on NVIDIA GPUs | vLLM, SGLang, or TensorRT-LLM | NVIDIA Dynamo | Dynamo router | Dynamo Planner |
| A quick start on AKS with model presets | vLLM | KAITO | Inference Extension | KEDA |
| A Python pipeline of several models | Any | Ray Serve on KubeRay | Ray | Ray autoscaler |
| An agent's small-model fleet (embeddings, reranking, extraction, small LLMs) | Chosen per model by SIE | SIE | SIE gateway | KEDA (included) |
| A large model and a small-model fleet | Both | llm-d or Dynamo, plus SIE | Separate | Separate |
Not sure which option to choose? Superlinked's llm-d vs Dynamo vs SIE comparison suggests a simple test: count how many distinct models one user request touches.
Don’t Build Everything Yourself
If the table points you to the small-model row, that’s the part I wouldn’t build from scratch.
The do-it-yourself version means one deployment per embedding model, reranker, and extractor, one autoscaler config each, a gateway in front of all of them, and GPUs that spend much of their time idle between requests.
SIE puts all that behind one API with four operations:
encode, score, extract, and generate.In Superlinked's write-up from the Infer() Summit, CEO Daniel Svonava reported more than 50% higher throughput on real small-model workloads from SIE's shared queue, which forms a full batch before the GPU runs.
To be clear about the limits, though:
- It’s not for one giant model. Use llm-d or Dynamo for that, and run SIE next to it.
generatetargets small open LLMs, not frontier-scale models.- Cold starts from zero take 3 to 7 minutes, so keep a warm replica for user-facing paths.
- It needs Kubernetes 1.29 or newer and NVIDIA GPU nodes for production.
If this sounds like a fit, the quickstart takes a few minutes. For more depth, read How to Serve a Fleet of Small Open-Source Models, How to Choose an Inference Layer for Agents, and Hundreds of Models, One Deployment.
Happy building!
FAQ
What is the best tool for serving open-source models on Kubernetes?
There’s really no single best tool. vLLM or SGLang is usually the runtime, and the platform depends on your workload: that is, whether you serve one large model (llm-d, KServe, Dynamo) or many small ones (SIE).
Is Hugging Face TGI still a good choice in 2026?
TGI has been in maintenance mode since December 2025, so new deployments should start on vLLM or SGLang.
Can GPU inference scale to zero on Kubernetes?
Yes, with KEDA or Knative, but expect cold starts of several minutes when a new GPU node needs to start.
Can I run SIE next to llm-d or NVIDIA Dynamo?
Yes, they run as separate control planes on separate GPU pools and communicate over HTTP.