InferenceOps: Building a Reproducible LLM Serving Lab¶

Introduction¶

What you'll learn¶

• How I built a serving benchmark from a single vLLM baseline to a two-node GPU cluster.
• Why deterministic trace replay matters for LLM inference evaluation.
• How SGLang Gateway and vLLM Production Stack fit into an inference operations workflow.
• What heuristic and agentic controllers can change, and what they cannot prove without strong baselines.
• What broke during real Kubernetes GPU deployment and how I fixed it.

Why I built this¶

Most LLM projects focus on model quality, prompts, retrieval, or agents. Those are important, but production LLM systems also live or die on the serving layer.

A serving system has to answer questions like:

• Can the same workload be replayed later with comparable results?
• Which routing policy handles repeated-prefix traffic better?
• What happens when one worker is slower or overloaded?
• Does adding a second GPU actually improve throughput?
• What does Kubernetes add once the model server becomes a production service?

InferenceOps was my attempt to make those questions testable in a small but realistic workspace.

System overview¶

The project has five core layers:

• Trace packs: deterministic workload definitions with prompts, arrival times, model metadata, prefix groups, and target output lengths.
• Replay harness: sends OpenAI-compatible requests into serving backends and records per-request results.
• Serving backends: vLLM, SGLang Gateway, and vLLM Production Stack.
• Controllers: static policies, heuristic policies, and a bounded optional agentic controller.
• Artifacts and metrics: normalized JSON/CSV outputs, summaries, run reports, and milestone recaps.

The important design decision was to separate the request/data plane from the control plane. OpenAI-compatible APIs are useful for sending requests, but backend-specific adapters are still needed for policy changes, worker health, metrics, drain/resume behavior, and unsupported operations.

Milestone roadmap¶

Milestone 0: vLLM baseline¶

The first step was a single-backend vLLM deployment on a Lambda GPU VM. This gave the project a known-good serving baseline before adding routers or controllers.

This milestone was intentionally simple. Before testing routers, controllers, or Kubernetes, I wanted one real model server producing reliable artifacts.

Milestone 1: deterministic trace replay¶

The next step was moving from fixed benchmark commands to deterministic trace generation and replay.

The first trace pack, traces/pack_v1, included three scenarios:

The trace schema stores more than token counts:

• prompt text or deterministic prompt-generation metadata
• tokenizer and model revision
• seed
• prefix group
• input and output targets
• request arrival time

That matters because token lengths alone are not enough to evaluate cache locality or repeated-prefix behavior.

The full trace pack replay on vLLM completed with zero failures:

Milestone 2: SGLang Gateway router¶

Milestone 2 added a real router layer using SGLang Gateway in front of two workers.

replay_trace_pack.py -> SGLang Gateway :30000
                         -> SGLang worker 1 :31001
                         -> SGLang worker 2 :31002

The benchmark compared three routing policies across the three trace scenarios:

• round_robin
• cache_aware
• power_of_two

All policy/scenario runs completed successfully: 432 total requests, 0 failures.

This was the first point where the project became more than a load generator. It could compare routing behavior under repeatable workloads.

Milestone 3: heuristic controller policies¶

Milestone 3 added controller logic on top of the routing benchmark. The controller action space was bounded:

set_policy in {round_robin, cache_aware, power_of_two}

Implemented controllers included:

• static_round_robin
• static_cache_aware
• static_power_of_two
• scenario_heuristic
• tail_guard

The first heuristic controller selected policies by scenario:

The live controller run completed 144 requests with 0 failures.

The key outcome was not that one heuristic is universally best. The important part was building a repeatable controller evaluation path with finite, auditable actions.

Milestone 4: bounded agentic controller¶

The agentic controller was intentionally introduced after heuristic baselines. It was not allowed to invent arbitrary infrastructure operations. It could only choose from the same bounded policy actions.

That framing matters. An agentic controller is interesting, but it should not be the first benchmark result. In this project, it was treated as an optional research layer after reproducible traces, real routers, and heuristic baselines were already working.

The useful lesson was architectural: agentic control needs a narrow action space, deterministic evaluation, and strong baselines. Without those, it is hard to tell whether the agent is improving the system or just adding complexity.

Milestone 5: vLLM Production Stack on k3s¶

Milestone 5 moved the benchmark onto vLLM Production Stack using k3s on a single GPU VM.

Deployment path:

Lambda GPU VM
  -> k3s
  -> NVIDIA container runtime
  -> NVIDIA Kubernetes device plugin
  -> vLLM Production Stack Helm chart
  -> vLLM router Kubernetes Service
  -> replay_trace_pack.py

This milestone added Kubernetes deployment, Helm configuration, GPU device plugin setup, router service access, and cluster-state collection.

This was valuable because it moved the project into an industry-style serving stack without immediately taking on the complexity of a full managed Kubernetes cluster.

Milestone 6: two-node GPU cluster¶

The final experiment used two Lambda GPU VMs as a k3s cluster:

• server/control-plane node: A100 40GB
• worker node: A100 40GB
• one vLLM model pod per GPU node
• vLLM router service in front of both model pods

The first attempt exposed a real Kubernetes storage issue. The vLLM Production Stack chart created one local PVC for the model deployment. With two replicas, the first pod bound the PVC to one node, and the second pod could not schedule on the other GPU node.

The fix was to define two one-replica model deployments, each with its own PVC. That allowed Kubernetes to place one vLLM model pod on each GPU node.

The two-GPU deployment worked, but this trace pack did not saturate the system enough to demonstrate throughput scaling. That is a useful benchmark finding: scaling experiments need workloads designed to create real pressure, not just successful replay.

Results snapshot¶

What broke and what I learned¶

Kubernetes API endpoint routing¶

The first two-node k3s install advertised the public IP as the API endpoint. That made internal cluster services unreliable. CoreDNS and metrics-server had trouble reaching the Kubernetes service endpoint.

The fix was to rebuild the cluster using Lambda private IPs for internal node and API routing, while keeping the public IP only for SSH/admin access.

Local PVC scheduling¶

The second issue was Kubernetes storage affinity. A local PVC is bound to the node where it is provisioned. One shared PVC across two GPU replicas meant the second replica could not schedule on the other node.

The fix was to avoid one shared local PVC for a multi-node deployment. In this experiment, two one-replica model specs gave each pod its own PVC and allowed one pod per GPU node.

Scaling requires saturation¶

The two-replica system worked, but the workload was not heavy enough to show throughput improvement. That is not a failed experiment. It is a benchmark-design result.

A stronger scaling test should include:

• higher concurrent arrivals
• longer output targets
• longer run duration
• GPU utilization sampling during replay
• router-level request distribution metrics

What I would build next¶

• Add a saturation trace pack with higher concurrency and longer output targets.
• Collect timestamped GPU utilization during replay.
• Compare router behavior under actual saturation, not just normal replay.
• Add a compact dashboard for run comparison.
• Test managed Kubernetes only if the goal shifts toward deeper platform/MLOps work.

Links¶

• Project repo: github.com/SulmanK/InferenceOps
• Design docs: InferenceOps design folder
• Trace pack: traces/pack_v1
• Deployment scripts: deploy